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ABSTRACT 

In an earlier pilot study of the narrative sections 
of Navy performance evaluations for senior enlisted personnel in pay 
grade E-7, it was determined by content analytic techniques that it 
is possible to differentiate between the performance of typical and 
superlative chief petty officers based on the narrative content of 
Evaluation Reports, A second study attempted to cross validate the 
pilot study results on new Evaluation Reports for senior enlisted men 
in the same two occupational ratings (AT 1 s and BT 1 s) represented in 
the pilot study sample and to extend the content analysis to two 
different occupational ratings (CS's and RM's) in order to test the 
generalizability of the content analytic techniques developed 
earlier. As a further refinement, the cross validation and 
generalization samples were analyzed without any knowledge of the 
ratee^s relative position in the upper half of the marking scale on 
Performance of Dutj (the criterion variable) • A reliability study was 
also conducted whose objectives were twofold: (1) to determine the 
level of agreement among four individuals all of whom independently 
would perform a content analysis of the same corpus of 48 Evaluation 
Reports, and (2) to investigate if nonresearchers could be trained 
successfully to apply the complex content analysis methodology 
developed in the pilot study. This report presents detailed evidence 
that the pilot study findings essentially were extendible to the 
blind cross validation sample consisting of the same two ratings and 
also were generalizable to two different occupational ratings. 
Despite the initial expectation that it would be extremely difficult 
to train nonresearch-oriented individuals to consistently index the 
narrative sections of Evaluation Reports, the surprising result in 
the reliability study was that in only six training sessions a quite 
respectable level of agreement was achieved, (Author) 
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SUMMARY OF. FINDINGS 

In an earlier pilot study of the narrative sections of Navy performance 
evaluations for senior enlisted personnel in pay grade E-7, it was determined 
by content analytic techniques that It is possible to differentiate between 
the performance of typical and superlative chief petty officers based on the 
substantive content of Evaluation Reports. The results of this pilot study 
strongly suggested that there are stable differences among the performance 
characteristics of chief petty officers in the various portions of the upper 
half of the marking scale on Performance of Duty that are reflected in nar- 
rative statements written by evaluators. 

The findings from the pilot study were considered to be provocative 
enough to warrant further investigation. Therefore, a second study was em- 
barked upon to attempt to cross validate the pilot study results on new Evalua 
tion Reports for senior enlisted men in the same two occupational ratings (AT f 
and B^s) that were represented in the pilot study sample and to extend the 
content analysis to Evaluation Reports for senior enlisted men in two differ- 
ent occupational ratings (CS's and RM's) than those investigated in the pilot 
study iii order to test the generalizability of the content analytic techniques 
developed earlier. As a further refinement, the cross validation and gener- 
alization samples of Evaluation Reports were to be analyzed without any knowl- 
edge of the ratee's relative position in the upper half of the marking scale 
on Performance of Duty (the criterion variable). In the pilot study the cri- 
terion data were made available early in the study, thus introducing the pos- 
sibility that this knowledge subconsciously might have influenced the content 
analysis that was performed. This factor was controlled for in the second 
study by withholding the criterion information until the content analysis of 
the narrative text had been completed. 

Also of concern in the pilot study were the issues of reliability and 
trainability , although the scope of the small initial research effort did not 
permit these aspects to be studied in any substantial way. Therefore, in de- 
signing the second investigation these issuer, were dealt with by including a 
reliability study whose objectives were twofold: (1) to determine the level 
of agreement among four individuals all of whom independently would perform a 
content analysis of the same corpus of 48 Evaluation Reports, and (2) to in- 
vestigate if nonresearchers could be trained successfully tc apply the complex 
content analysis methodology developed in the pilot study. 

In the earlier pilot investigation, the Navy Personnel Research and 
Development Center (NPRDC), San Diego, selected a sample of 225 Evaluation Re- 
ports for senior enlisted personnel in pay grade E-7 including 145 Aviation 
Electronics Technicians (AT's) and 80 Boilermen (BT's). All 225 Evaluation 
Reports were drawn from the top half of the marking scale on 19A-PERF0RMANCE 
OF DUTY of Evaluation Report Form NAVPERS 1616/8. This form subsequently ? has 
been replaced by another form that can be scanned by an optical character 
reader; however, the content of the two forms is essentially the same. The 
pilot study sample of 225 Evaluation Reports was divided equally into three 
criterion groups Upper, Middle, and Lower corresponding to three continu- 
ous segments of the upper half of the marking scale on 1 9 A-P ERFORMANC E OF 
DUTY. 
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In the study being reported here, NPRDC also selected the sample of Evalua 
tion Reports to be analyzed. The cross validation sample consisted of 222 
Evaluation Reports from the same two ratings that were used in the pilot study 
(i.e., AT T s and BT ! s). In addition, a generalization sample consisting of 222 
Evaluation Reports was also selected by NPRDC from two different ratings in 
order to ascertain the generalizability of the content analytic methodology 
developed in the pilot study. The two ratings from yhich the generalization 
sample was drawn were Commissary-men (CS T s) and Radiomen (RM T s). The N T s in 
the various occupational ratings represented in the cross validation and gen- 
eralization samples were 138 AT T s , 84 BT T s, 60 CS T s, and 162 RM T s .• The cross 
validation sample and the generalization sample were each divided equally into 

the same three criterion groups as the pilot study sample Upper, Middle, and 

Lower. Actual criterion group membership for the cross validation sample and 
the generalization sample was known only to NPRDC until the content analysis 
of the narrative text had been completed. Consequently, the content analysis 
of these two samples was conducted in the blind without benefit of knowing to 
which criterion group each Evaluation Report belonged. 

An indexing vocabulary consisting of 29 descriptive labels was devised to 
encompass the substantive content of the narrative sections of Evaluation Re- 
ports. These 29 index terms fell into three major areas MANAGEMENT FUNCTIONS 

SKILLS AND ABILITIES, and PRODUCTIVITY AND ACHIEVEMENT. Under each of these 
headings there were more detailed terms such as PLANNING, TECHNICAL SKILLS, 
and AWARDS AND PUNISHMENT, providing the indexer with a 3-level hierarchy of 
descriptive labels from which to choose. Each sentence of narrative text in 
the pilot study sample and the cross validation and generalization samples was 
read carefully and, where appropriate, divided into segments corresponding to 
the assignment of specific index terms. However, it is not enough to simply 
label a narrative statement with the most appropriate index term since the 
statement may have been a highly positive, quite positive, neutral, quite nega- 
tive, or highly negative one. Therefore, a weighting scale containing five 
degrees of favorableness/unf avorableness was devised based on the range of ad- 
jectives and adverbs that occur in narrative text of this kind. The indexing 
procedure that was used in this study was the following: The narrative text 
of each Evaluation Report was read, segmented into distinct statements, and 
each statement was assigned one or more index terms from the set of 29 pos- 
sible choices. Each term selected was also assigned a numerical weight from 
1 to 5 depending upon the nature of the adjectives or adverbs used as modi- 
fiers in the statement. When the entire narrative text of the Evaluation Re- 
port had been indexed, the indexing decisions that had been made were record- 
ed on a special indexing form. 

A set of 67 quantitative variables was derived from the indexing form 
used in the content analysis. The first 29 variables reflect the simple fre- 
quency with which each index term was used to index a particular section of 
narrative text. Variable 30 is the sum of these 29 frequencies. Variables 
30 through 59 represent the weighted frequency of each index term used to 
index a particular section of narrative text. Variable 60 is similar to Varia- 
ble 30 in that it is the sum of the 29 weighted frequencies. Variables 61 
through. 65 represent the frequency counts over the entire indexing form for 
all 5 weights, 4 weights, 3 weights, 2 weights, and 1 weights. Variable 66 is 
the total number of words in the section of narrative text that was indexed. 
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Variable 57 is the total number of index terms of the 29 available that were 
used to index the section of narrative text . Profiles or vectors of these 
67 values then were prepared for all of the Evaluation Reports contained in 
each sample. Separate profiles were compiled for the evaluation section (19R) 
and the justification section (19S) of each Evaluation Report. 

Descriptive statistics were computed for each of the three research sam- 
ples on the 67 quantitative variables. As expected these statistics showed 
in general that the higher the criterion group, the longer the narrative text. 
Also, as the evaluator uses more words to describe the ratee, he is more like- 
ly to comment on a wider variety of specific areas of the ratee' s performance. 
Correlations among all 67 variables for the evaluation section and for the 
justification section also were computed for the cross validation and generali- 
zation samples as well as the matrix correlating the evaluation section with 
the justification section on all 67 variables. There were very few high cor- 
relations except for the correlations between Variables 1 through 30 and 
Variables 31 through 60, these two sets of variables being the same except for 
the method of weighting that was used. Variable 67 (Total Number of Index 
Terms Used) is a focal variable, correlating highly with Variable 30 (Sum of 
Variables 1 through 29), Variable 60 (Sum of Variables 31 through 59), Varia- 
ble 61 (Total Number of 5 Weights), Variable 62 (Total Number of 4 Weights), 
Variable 63 (Total Number of 3 Weights), and Variable 66 (Total Number of 
Words in the Narrative Text). Correlations among the variables having to do 
with the 29 index terms per se were notably low, indicating that these 29 di- 
mensions are relatively orthogonal- and represent independent aspects of mana- 
gerial performance. In the correlation matrix for the evaluation section ver- 
sus the justification section, no high correlations were evident in either the 
cross validation sample or t]je generalization sample, demonstrating that these 
two narrative sections of Evaluation Report Form NAVPERS 1616/8 are quite in- 
dependent and should be treated separately. 

Both the Mann-Whitney U test and the t test of mean difference were com- 
puted on each of the 67 variables for the pilot study sample, the cross valida- 
tion sample, and the generalization sample. These computations were made for 
each pair of criterion groups in both the evaluation and the justification sec- 
tions. The most difficult discrimination to be made is that between the Mid- 
dle and Upper criterion groups. In the cross validation sample the character- 
istics that differentiated outstanding CPO's from their slightly less quali- 
fied colleagues on the evaluation section were cooperation, grooming and at- 
tire, resourcefulness, and productivity and achievement. Cooperation was also 
. implicated as a discriminating variable between the Middle and Upper criterion 
groups on the evaluation section in the pilot study sample. On the justifica- 
tion section, eight variables showed a statistically significant difference 
between the Middle and Upper criterion groups in both the pilot study sample 
and the cross validation sample. When an evaluator is required to justify 
his marks in evaluating a ratee, he apparently calls out certain areas of 
performance that distinguish the ratee in the Upper^criterion group from his 
slightly less qualified colleague in the Middle criterion group. Skills and 
abilities as well as productivity and achievement were the differentiating 
areas of performance. The ratee in the Upper criterion group also had be- 
stowed upon him more superlative adjectives and adverbs. Total Number of Words 
in Text and Total Number of Index Terms Used were also discriminating variables. 
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In the comparison between the Middle and Upper criterion groups on the 
evaluation section of the generalization sample, Total Number of 5 Weights 
(Excellent), Total Number of 2 Weights (Poor), and potential were the discrimi- 
nating variables. None of these variables overlapped with those that were 
statistically significant for the Middle versus Upper criterion group compari- 
son on the evaluation section of the cross validation sample. Seventeen of 
the 67 quantitative variables showed a statistically significant difference be- 
tween the Upper and Middle criterion groups on the justification section for 
both the cross validation sample and the generalization sample, the most sig- 
nificant of which were Total Number of 5 Weights (Excellent), Total Number of 
4 Weights (Good), Total Number of 2 Weights (Poor)^ Total Number of Index 
Terms Used, Sum of Variables 1 through 29, Sum of Variables 31 through 59, 
leadership and directing, communication, cooperation, technical skills, drive, 
and potential. This finding suggests that there is partial overlap between 
these two samples. The nonoverlapping areas may be attributed to a difference 
in the nature of the occupational ratings represented in the cross validation 
and the generalization samples and/or to unreliability in the indexing pro- 
cedure, although the results of the reliability study suggest that differences 
among the four occupational ratings provide a more reasonable explanation for 
these results than unreliability. 

In every pairwise criterion group comparison made for the three research 
samples using the Mann-Whitney U test and the t test of mean difference, one 
or more of the five variables involving total number of weights proved to be 
significantly differentiating. This finding supports the results of the cor- 
relational analysis in which the system used for weighting Variables 31 through 
59 was highlighted as an important aspect of the content analyses methodology. 

It is of considerable interest to learn how well the set of 67 quantita- 
tive variables, used in optimal combination, can classify each of the research 
samples into correct' criterion group. A stepwise discriminant analysis pro- 
gram was used to perform this analysis. A special feature of this program 
allows new cases to be classified by the discriminant functions generated on 
the original sample. This feature was used to conduct two cross validation 
studies of the AT ! s and BT's combined and also of the AT ? s and BT's considered 
separately. The results of the two cross validation studies were very similar 
for the total cross validation sample and the total pilot study sample, for 
the cross validation AT's and the pilot study AT's, and for the cross valida- 
tion BT's and the pilot study BT's. These findings support the expectation 
held at the outset of this investigation that it would be possible to index 
the cross validation sample in the blind, without knowledge of criterion group 
membership, and achieve as good classification accuracy as was achieved with 
the pilot study sample where criterion group membership was known to the in- 
dexer. Further, it can be concluded that better classification into the 
three criterion groups using r.a optimum combination of the 67 quantitative 
variables is achieved when the two occupational ratings represented in the pi- 
lot study sample and the cross validation sample are treated separately. 
These findings suggest that classification procedures based on the content 
analysis methodology developed in this research should be tailored to specific 
occupations. In all of the stepwise discriminant analyses performed, better 
classification was achieved in the analysis of the justification section com- 
pared to the evaluation section. Classification of each sample by its own 
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discriminant functions achieved perfect classification for the BT f s on the 
justification section in both the pilot study sample and the cross validation 
sample. In the pilot study sample, 95 percent of the AT T s were classified 
correctly on the justification section. In the cross validation sample, 93 
percent of the AT's were classified correctly on the justification section. 
The superior classification accuracy achieved for the BT T s compared to the 
AT f s indicates that the Aviation Electronics Technician rating may represent 
a more varied amalgamation of technical activities than the Boilerman rating. 
Further, it appears that the best classification accuracy than can be achieved 
on a second sample using the discriminant functions generated on the first 
sampxe, with the content analysis methodology developed thus far, is 65 to 70 
percent. 

In the stepwise discriminant analyses of the generalization sample, all 
60 generalization CS T s were correctly classified on the justification section . 
Of the 162 generalization RM T s, 39 percent were correctly classified on the 
justification section. This suggests that the occupational rating, Radioman, 
may be more heterogeneous and require a greater variety of skills than the 
Commissaryman rating just as the Aviation Electronics Technician rating may 
represent a more varied amalgamation of technical activities than the Boiler- 
man rating. It is evident from these results that the content analysis method- 
ology developed initially on the pilot study sample consisting of ' AT's and 
BT T s was generalizable to a new sample consisting of two different occupation- 
al ratings, viz., CS f s and RM T s. 

In the double cross validation of the pilot study sample and the cross 
validation sample, the best classification accuracy for the sample being 
cross validated was achieved early in the stepwise discriminant analysis pro- 
cedure, typically by the fifth step. The key discriminating variables for 
the evaluation section were Total Number of 5 Weights (Excellent) and Total 
Number of 2 Weights (Poor), In the justification section without exception 
the key discriminating variable was Total Number of Index Terms Used. These 
same three variables were those selected first in the stepwise discriminant 
analysis of the generalization sample. It appears that the modifying adjec- 
tives used by an evaluator to rate a ratee and the range of skills and abili- 
ties that a chief petty officer possesses may be key factors in the ratee's 
superior performance. The results also suggest that a smaller number of di- 
mensions than the full complement of 67 quantitative variables derived from 
the indexing procedure can be used to identify superlative CP0 T s whose supe- 
rior performance recommends them as candidates for promotion to a higher level 
of responsibility. 

In addition to the cross validation and generalization study, a compre- 
hensive reliability study was conducted whose objectives were twofold: (1) 
to determine the level of agreement among several individuals all of whom 
independently would perform a content analysis of the same corpus of Evalua- 
tion Reports, and (2) to investigate if nonresearchers could be trained suc- 
cessfully to apply the complex content analysis methodology developed in the 
pilot study. 

A set of 48 Evaluation Reports was selected by the Navy Personnel Research 
and Development Center, representing a cross section of the kinds of reports 
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included in the overall experimental design C^r the cross validation and gener- 
alization samples. In each of these 48 Evaluation Reports the evaluation sec- 
tion was separated from the justification section so that the narrative comments 
for each section were not considered together. This resulted in a group of 96 
randomized pieces of narrative text to be indexed in the reliability study. 

Four individuals participated in the reliability study: (1) the experi- 
enced indexer who also indexed the pilot study sample, the cross validation 
sample, and the generalization sample; (2) the principal investigator; (3) an 
inexperienced indexer (inexperienced indexer A) with two year. 3 of college in 
the liberal arts; and (4) another inexperienced indexer (inexperienced indexer 
B) with executive secretary experience. To this end a training manual was 
prepared by the experienced indexer and the principal investigator to assist 
the two neophyte indexers in understanding their assignment. Six intensive 
training sessions were conducted by the experienced indexer in order to try 
to bring all four indexers up to a common level of expertise before beginning 
the actual study. Obviously, this objective could only be met partially in 
view of the varying educational backgrounds of the four reliability indexers 
and their different levels of previous exposure to the indexing dictionary. 

In all of the agreement statistics that were computed, assignment of the 
index terms was considered to be a separate intellectual task from assigning 
the corresponding weights based on the modifying adjectives and adverbs. The 
kappa statistic was the measure of agreement used in analyzing the index terms 
assigned by the four reliability indexers. The best agreement in selecting 
index terms was obtained between the experienced indexer and inexperienced in^ 
dexer A, a kappa of ,88 where the maximum kappa possible in this instance was 
.97. Of the six possible pairwise comparisons between the four reliability 
indexers, the value of kappa ranged from ,71 to .88, with ,71 probably rep- 
resenting the lower limit of reliability achievable in a study of this kind, 
The kappa analysis revealed that the major area of confusion in indexing the 
reliability data base resided in whether or not to index supposedly factual 
statements describing the job duties and the qualifications needed for the 
position that the ratee occupied rather than the ratee 1 s actual performance in 
this position. All three of the less experienced indexers tended to index 
these statements as describing the ratee 1 s performance whereas the experienced 
indexer whom the other three indexers were trying to emulate treated these 
statements as factual descriptions of the job duties and the qualifications 
needed for the position. Additional training aimed at clarifying this area 
of confusion most likely would markedly reduce this type of disagreement and 
raise the magnitude of kappa. 

Analysis of the level of agreement among the four reliability indexers in 
assigning numerical weights to each index term selected, based on the modify- 
ing adjectives and adverbs, was performed differently than the analysis of the 
level of agreement in selecting the index terms themselves, because the numeri- 
cal weights assigned to the index terms constituted an ordinal scale whereas 
the index terms themselves formed a nominal scale. In the six pairwise com- 
parisons between the four -reliability indexer?, six product moment correlation 
coefficients were computed as well as another agreement statistic, weighted 
kappa, in order to determine if weighted kappa agreed with the results of the 
correlational analysis. In the correlational analysis, once again the best 
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agreement in assigning numerical weights to each index term selected was obtained 
between the experienced indexer and inexperienced indexer A, a correlation co- 
efficient of .80, These findings corroborate each other in suggesting that an 
individual without a research background in only six training sessions can be 
taught not only how to select the most appropriate index terms but also how to 
consistently assign weights to these terms based on the modifying adjectives 
and adverbs. The other five correlation coefficients were lower, but none 
less than .64. 

As was expected, the weighted kappa values were similar in magnitude to 
their correlation coefficient counterparts. Again, the best agreement as 
measured by weighted kappa was obtained between the experienced indexer and in- 
experienced indexer A, a weighted kappa of ,78, If the area of confusion in- 
volving overindexing on the part of inexperienced indexer A was ignored in the 
analysis, the value of weighted kappa increased to .82. The gain in the value 
of Xfeighted kappa is not very large for the comparison between the experienced 
indexer and inexperienced indexer A when weighted kappa was recomputed in this 
fashion. However, the gain was quite substantial in the other comparisons be- 
tween the experienced indexer and the principal investigator and between the 
experienced indexer and inexperienced indexer B, This suggests that with ad- 
ditional training to clarify this area of confusion and with more indexing 
experience, the level of agreement among the four reliability indexers could 
possibly be raised to a value of .80 to ,85 as measured by any of the three 
agreement statistics employed in this study. However, values in the .90 T s 
are the ultimate objective. 

In conclusion, it might be of interest to point out that the initial ex- 
pectation in beginning this reliability study was that it would be extremely 
difficult to train nonre^earch-qriented individuals to consistently index the 
narrative sections of Evaluation Report forms using the complex content analy- 
sis methodology that had oeen developed in the pilot study. The surprising 
result is that in only six training sessions a quite respectable level of 
agreement was achieved. Moreover, one of the inexperienced indexers showed 
a higher level of agreement with the experienced indexer than the principal 
investigator did, and the other inexperienced indexer agreed with the experi- 
enced indexer almost as well as the principal investigator. The intuitive 
feeling that the reliability indexers had after completing the reliability 
study was that the most difficult part of learning to index consistently was 
over and that with additional practice and some review training sessions they 
could improve their indexing skill. 
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A goal of on-going research being conducted by the Navy Personnel Re- 
search and Development Center, San Diego, is to develop Navy enlisted perform- 
ance evaluation formats which will be effective in holding down the pile-up of 
marks at the high end of the marking scale and in achieving a distribution of 
marks which tapers off sufficiently at the high end of the scale to permit 
greater differentiation iirong ratees, making evaluations more useful, especial- 
ly when small selection opportunities are involved. 1 Thus far the narrative 
sections of Evaluation Reports have not been exploited to any great extent in 
the design of experimental forms because narrative text tends to resist easy . 
analysis. However, in a pilot investigation of the narrative sections of Navy 
performance evaluations for senior enlisted personnel in pay grade E-7 con- 
ducted by R-K Research and System Design, it was determined by content analytic 
techniques that it is possible to differentiate between the performance of 
typical and superlative chief petty officers based on the substantive content 
of Evaluation Reports* 2 The result? of this pilot study strongly suggested 
that there are stable differences among the performance characteristics of 
chief petty officers in the various portions of the upper half of the marking 
scale on Performance of Duty that arr^ reflected in narrative statements writ- 
ten by evaluators. These differences are both identifiable and quantifiable. 
In the pilot study the significant differences resided in the superiority of 
the uppermost criterion group with respect to managerial skills and abilities, 
particularly as demonstrated in the areas of organization, initiative, coop- 
eration, leadership and directing, professionalism, productivity and achieve- 
ment, more awards and fewer punishments, more drive, more superlative attri- 
butes, and fewer qualified statements reflecting fair performance. 

The findings from the pilot study were considered to be provocative 
enough to warrant further investigation. Therefore, a second study was em- 
barked upon to attempt to cross validate the pilot stud) results on new Evalua- 
tion Reports for senior enlisted men in the same two occupational ratings that 
were represented in the pilot study sample and to extend the content analysis 
to Evaluation Reports for senior enlisted men in two different occupational 
ratings than those investigated in the pilot study in order to test the gener- 
alizability of the content analytic techniques developed earlier. As a fur- 
ther refinement, the cross validation and generalization samples of Evaluation 
Reports were to be analyzed without any knowledge of the ratee T s relative posi- 
tion in ,the upper half of the marking scale on Performance of Duty (the crite- 
rion variable). In the pilot study the criterion data were made available 
early in the study, thus introducing the possibility that this knowledge sub- 
consciously might have influenced the content analysis that was performed on 
the narrative sections of the Evaluation Reports. This factor was controlled 
for in the second study by withholding the criterion information until the con- 
tent analysis of the narrative text had been completed. 

Also of concern in the pilot study were the issues of reliability and 
trainability , although the scope of the small initial research effort did not 
permit these aspects to be studied in any substantial way. Therefore, in de- 
signing the second investigation these issues were dealt with by including a 
reliability study whose objectives were twofold: (1) to determine the level 
of agreement among four individuals all of whom independently would perform a 
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content analysis of the same corpus of Evaluation Reports, and (2) to investi- 
gate if nonresearchers could be trained successfully to apply the complex con- 
tent analysis methodology developed in the pilot study. 

The second study attempting to cross validate and generalize the pilot 
study results and to elucidate the issues of reliability and trainability was 
conducted during the contract year March 1, 1972 to February 28, 1973. This 
technical report presents the findings resulting from this follow-on investi- 
gation. Section 2 of this report describes the nature of the cross validation 
and generalization samples. Section 3 presents the content analysis methodolo- 
gy that was used in both the pilot study and the second study. Section 4 dis- 
cusses the statistical methodology that was used to analyze the data and the 
results that were obtained. Section 5 describes the design of the reliability 
study and presents the results achieved. In Section 6 future areas of inves- 
tigation are delineated. 
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SECTION 2. NATURE OF THE PILOT STUDY SAMPLE AND THE CROSS VALIDATION 
AND GENERALIZATION SAMPLES 

As a result of research conducted at the Navy Personnel Research and De- 
velopment Center, San Diego, to develop experimental forms for evaluating per- 
sonnel- in pay grades E-7 (Chief Petty Officer), E-8 (Senior Chief Petty Offi- 
cer), and E-9 (Master Chief Petty Officer), a new evaluation report form 

NAVPERS 1616/8 was introduced into operational use in January 1969 (see Fig- 
ure 1),* This form had been demonstrated as effecting a substantial improve- 
ment in the distribution of operational evaluation marks over the previously 
used form, NAVPERS 792, as reflected in a reduced pile-up of marks at the high 
end of the marking scale and greater differentiation among ratees. 1 

Section 19, Evaluation Section, of Evaluation Report Form NAVPERS 1616/8 
is designed to permit the rater (evaluator) to compare the ratee with all 
others of his rate known to the rater on 13 specific aspects of on-job perform- 
ance. Ratings are made by marking the column of the rating distribution into 
which the rater evaluates that the ratee falls for each of the 13 specific as- 
pects of on-job performance plus an overall evaluation of the ratee (for exam- 
ple, top 1% for superlative performance). Section 19R of this form provides 
space for the rater to write narrative evaluation comments to describe further 
the ratee 1 s performance and qualifications. Section 19S of this form provides 
space for the rater to write narrative justification comments and is required 
to support any marks assigned to the top or bottom 10, 5, or 1% columns of Sec- 
tion 19. 

Sections 19R and 19S are referred to as the narrative text of the Evalua- 
tion Report since they are the only portions of the report where the rater 
uses his own words to assess the on- job performance of the senior enlisted man 
that he is rating. Thus far the narrative evaluation and justification sec- 
tions of the Evaluation Report have not been exploited systematically in mak- 
ing personnel decisions because narrative text tends to resist objective analy- 
sis and interpretation. 

In the earlier pilot investigation, the Navy Personnel Research and De- 
velopment Center (NPRDC) , San Diego, selected a sample of 225 Evaluation Re- 
ports for senior enlisted personnel in pay grade E-7 taken from a pool of ap- 
proximately 1,000 performance evaluation report forms for two occupational 

ratings Aviation Electronics Technician (AT) and Boilerman (BT) . All 225 

Evaluation Reports were drawn from the top half of the marking scale on 19A- 
PERFORMANCE OF DUTY located in the upper right quadrant of Evaluation Report 
Form NAVPERS 1616/8, The 19A-PERF0RMANCE OF DUTY category was used in pref- 
erence to 19N-0VERALL EVALUATION because standard scores (T Scores) were avail- 
able only for 19A. The use of standard scores rather than raw marks permitted 
a more refined selection to be made of the three criterion groups used in the 
study. Since raw marks on 19A correlate very highly with raw marks on 19N, it 
was felt that little was sacrificed by not using the overall evaluation and 



This form subsequently has been replaced by another form that can be scanned 
by an optical character reader; however, the content of the two forms is 
essentially the same. 
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that much was gained by using the purified T Scores on 19A. Only those Evalua 
tion Reports from commands spreading their marks and submitting eight or more 
E-7 and E-8 reports were considered. 

The pilot study sample of 225 Evaluation Reports was divided equally into 

three criterion groups Upper, Middle, and Lower corresponding to three con 

tinuous segments of the upper half of the marking scale on 1 9A-P ERF ORMAN C E OF 
DUTY. Table 1 shows the range of raw marks on 19A for each of the three cri- 
terion groups in the pilot study sample as well as the range and mean of T 
Scores. These standardized scores have a mean of 50 and a standard deviation 
of 10. Standardization was accomplished by setting each unit command mean 
equal to 50 and standardizing the total of E-7 and E-8 marks for each unit 
command. No cases from the bottom half of the marking scale on 19A were in- 
cluded in this study since there is no difficulty in differentiating these 
cases from the better performing personnel. 

Table 2 shows the distribution of the 225 pilot study Evaluation Reports 
among the three criterion groups for each of the two occupational ratings and 
for both occupations combined. After the pilot study sample had been selected 
and analyzed, it was discovered that one Evaluation Report for an Aviation 
Antisubmarine Warfare Operator (AW) had erroneously been coded as an Aviation 
Electronics Technician (AT). This case was removed from the analysis by spe- 
cific occupation but was left in the analysis for the total pilot study sample 

In the second study, NPRDC also selected the sample of Evaluation Reports 
to be analyzed. The same general procedures described above for selecting the 
pilot study sample were followed also in selecting the cross validation sample 
and the generalization sample, except that the i'orms were selected from a 



TABLE 1 

RANGE OF RAW MARKS, RANGE OF T SCORES, AND 
MEAN OF T SCORES ON 19A-PERF0RMANCE OF DUTY 
FOR THE THREE CRITERION GROUPS IN THE PILOT STUDY SAMPLE 



Criterion 
Group 


Range of 
Raw Marks 


Range of 
T Scores 


Mean of ! 
T Scores 


Upper 


In the top 5% 
column or the top 
1% column 


59.3 to 74.2 


64.68 


Middle 


In the top 10% 
column only 


48.0 to 54.1 


51.79 


Lower 


In the top 50% 
column or the top 
30% column 


33.8 to 39.7 


38.85 
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TABLE 2 

DISTRIBUTION OF THE 225 PILOT STUDY EVALUATION REPORTS 

AMONG THE THREE CRITERION GROUPS FOR EACH OF THE TWO. 
OCCUPATIONAL RATINGS AND FOR BOTH OCCUPATIONS COMBINED 



Occupational 
Rating 


Criterion Group 


Total 

N 


Upper 


Middle 


Lower 


AT 


49 


39 


56 


144 


* 










AW 


0 


i 


0 


1 


BT 


26 


35 


19 


80 


Total Sample 


75 


75 


75 


225 



This case erroneously was coded as an AT initially. 



subsequent year's data pool. The cross validation sample consisted of 222 
Evaluation Reports from the same two ratings that were used in the pilot study, 
that is, Aviation Electronics Technician (AT) and Boilerman (BT). In addition, 
a generalization sample consisting of 222 Evaluation Reports was selected by 
NPRDC from two different ratings in order to ascertain the generalizability of 
the content analytic methodology developed in the pilot study. The two rat- 
ings from which the generalization sample was drawn were Coramissaryman (CS) 
and Radioman (KM). 

The cross validation sample of 222 Evaluation Reports and the generaliza- 
zation sample of 222 Evaluation Reports were both divided equally into the 

same three criterion groups as the pilot study sample Upper, Middle, and 

Lower. Table 3 shows the range of raw marks on 19A for each of the three cri- 
terion groups in the cross validation sample (AT's and BT^s), the range of T 
Scores, and the mean of the T Scores for each criterion group. The£3 same 
data for the generalization sample (CS T s and RM's) are presented in Table 4. 
Tables 5 and 6 show the distribution of the 222 cross validation sample Evalua- 
tion Reports and the 222 generalization sample Evaluation Reports among the 
three criterion groups for each of the two occupational ratings represented in 
each sample and for both occupations combined. Actual criterion group member- 
ship for the cross validation sample and the generalization sample was known 
only to NPRDC until the content analysis of the narrative text had been com- 
pleted. Consequently, the content analysis of these two samples was conducted 
in the blind without benefit of knowing to which criterion group each Evalua- 
tion Report belonged. 
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TABLE 3 



RANGE OF RAW MARKS, RANGE OF T SCORES, AND 
MEAN OF T SCORES ON 1 9 A-P ERF ORMANC E OF DUTY 
FOR THE THREE CRITERION GROUPS IN THE CROSS VALIDATION SAMPLE 



Criterion 
Group 


Range of 
Raw Marks 


Range of 
T Scores 


Mean of 
T Scores 


Upper 


In the top 5% 
column or the top 

\ *y r 1 n 1 i TTtrn 


61,2 to 71,9 


64.23 


Middle 


In the top 10% 
column only 


48.2 to 55.9 


52.54 


Lower 


In the top 50% 
column or the top 
30% column 


30.3 to 42.0 


38.48 




TABLE 4 






RANGE OF RAW MARKS, RANGE OF T SCORES, AND 
MEAN OF T SCORES ON 19A-P ERFORMANCE OF DUTY 
FOR THE THREE CRITERION GROUPS IN THE GENERALIZATION SAMPLE 


Criterion 
Group 


Range of 
Raw Marks 


Range of 
T Scores 


Mean of 
T Scores 


Upper 


In the top 5% 
column or the top 
1% column 


61.2 to 74.8 


64.33 


Middle 


In the top 10% 
column only 


48.2 to 56,2 


52*50 


Lower 


In the top 50% 
column or the top 
30% column 


34.5 to 41.5 


38.56 



TABLE 5 



DISTRIBUTION OF THE 222 CROSS VALIDATION SAMPLE 
EVALUATION REPORTS AMONG THE THREE CRITERION GROUPS 
FOR EACH OF THE TWO OCCUPATIONAL RATINGS 
AND FOR BOTH OCCUPATIONS COMBINED 



Occupational 
Rating 


Criterion Group 


Total 


Upper 


Middle 


Lower 


N 


AT 


45 


44 


49 


138 


BT 


29 


30 


25 


84 


Total Sample 


74 


74 


74 


222 



TABLE 6 

DISTRIBUTION OF THE 222 GENERALIZATION SAMPLE 
EVALUATION REPORTS AMONG THE THREE CRITERION GROUPS 
FOR EACH OF THE TWO OCCUPATIONAL RATINGS 
AND FOR BOTH OCCUPATIONS COMBINED 



Occupational 
Rating 


Criterion Group 


Total 


Upper 


Middle 


Lower 


N 


CS 


19 


16 


25 


60 


RM 


55 


58 


49 


162 


Total Sample 


74 


74 


74 


222 
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SECTION 3. CONTENT ANALYSIS METHODOLOGY 

Conceptual Approach 

In the pilot study, the narrative portions of the 75 Evaluation Reports 
for each of the three criterion groups were read in their entirety before for- 
malizing the method of content analysis to be used. In this review the evalua- 
tion section and the justification section (19R and 19S) were considered sepa- 
rately. Borrowing from the field of information science, it seemed most ap- 
propriate to regard each narrative section as a short document that had been 
written by the ratee 's senior officer in order to communicate to a selection 
board or to a detailer the potential that the ratee had for promotion and in- 
creased responsibility. Considered in this framework, the analysis task then 
becomes one of ascertaining what the document is about (content analysis), 
specification of the content by a set of descriptive labels (indexing), and 
organization of an indexing vocabulary (controlling the form and semantics of 
the descriptive labels by lexicon and/or rule). 3,i * In order for the content 
analysis to be valid, Fairthorne 5 cautions that two aspects mukt be taken into 
consideration: (a) what the document is about, and (b) the circumstances of 
the expected uses of the content analysis with respect to a particular task or 
problem. Fairthorne's advice was attended to in the design of the content 
analysis methodology in that the indexing vocabulary which was developed re- 
lates strongly to the ultimate use to which performance evaluations are put, 
that is, the selection for promotion of outstanding chief petty officers in 
the face of limited promotional opportunities. 

The Indexing Vocabulary 

In reading the narrative portions of the 75 Evaluation Reports for each 
of the three criterion groups in the pilot study sample, it became apparent 
that the attributes and characteristics being evaluated for a ratee related 
primarily to his potential as a manager and supervisor. Consequently? five 
references in the area of managerial behavior and practice 6 5 7 5 ° 9 ^ 9 1 ^ were con- 
sulted as an aid to the development of the indexing vocabulary used in this 
study. An initial vocabulary containing 41 descriptive labels was devised and 
used to test the adequacy and manageability of the indexing method on 20 Evalu- 
ation Reports not included in the pilot study sample but similar to them in 
content. As a result of this experience, the original set of 41 labels was 
condensed into a more generic set of 29 index terms. The indexing form, in- 
corporating the final vocabulary that was used in both the pilot study and the 
second study, is shown in Figure 2. 

The top line of the indexing form carries fields for an identifying num- 
ber for each ratee, which criterion group he belongs to (used only in the pi- 
lot study since criterion data were withheld in the second study un£il the in- 
dexing had been completed), and whether the section being indexed is an evalua- 
tion section (19R) or a justification section (19S). The indexing form itself 
is divided into three major parts: MANAGEMENT FUNCTIONS, SKILLS AND ABILITIES, 
and PRODUCTIVITY AND ACHIEVEMENT. Under each of these headings there are more 
detailed terms, providing the indexer with a 3-level hierarchy of descriptive 
labels from which to choose. 
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ID No. Criterion Group Section 

Index Term 

MANAGEMENT FUNCTIONS 

CONTROLL ING 

LEADERSHIP AND DIRECTING 

ORGANIZATION 

PLANNING " 

REPRESENTATION 

STAFFING 

USE OF COMMUNICATION 

SKILLS AND ABILITIES 

COMMUNICATION 

CONDUCT, INTEGRITY, AND PRIDE 

COOPERATION 

ENDURANCE . 

FLEXIBILITY^ 

GROOMING AND ATTIRE 

INITIATIVE, 

INTELLECTUAL FUNCTIONING 

PROFESSIONALISM 

RELIABILITY AND D EP END AB IL I TY_ 

RESOURCEFULNESS 

RESPONSIVENESS 

TECHNICAL SKILLS 

PRODUCTIVITY AND ACHIEVEMENT 

AWARDS AND PUNISHMENT 

DRIV E 

SERVICE MOTIVATION 

* POT ENT I AL_ 

REPUTE 

ASSET TO THE NAVY 

FREQUENCY COUNTS s 5 , 4 , 3 , 2 , 1 _ 

TOTAL NUMBER OF WORDS TOTAL NUMBER OF INDEX TERMS 

Figure 2. Indexing Form Used in Performing the Content Analysis 



The first section of the indexing form includes seven specific MANAGEMENT 
FUNCTIONS that many authorities on management practice agree are the character- 
istic duties of all managers. 6,75 8,9,10 Although some authorities believe that 
there are more, less, or different functions performed by managers, these seven 
functions were selected because they are representative of the duties that 
chief petty officers actually perform. 

The second section of the indexing form contains index terras for 13 spe- 
cific SKILLS AND ABILITIES considered to be important by Navy supervisory per- 
sonnel in performing effectively as a chief petty officer. While some authori- 
ties on management practice consider making a judgment about whether or not an 
individual possesses a skill, quality, or ability to be a subjective process, 
Navy evaluators do repeatedly call out these specific qualities in their narra- 
tive evaluations because many of these qualities are dimensions on which they 
rate the ratee in Section 19 of the Evaluation Report. The first section of 

the indexing form MANAGEMENT FUNCTIONS deals with how a ratee performs his 

managerial functions and is result oriented, while the second section SKILLS 

AND ABILITIES contains index terms that relate to an individual's character- 
istics and qualities which, if used, may help him achieve good results. 

The third section of the indexing form PRODUCTIVITY AND ACHIEVEMENT 

is the most result-oriented section of the indexing hierarchy. Here are in- 
cluded the measures of overall performance, DRIVE and SERVICE MOTIVATION (a 
specific type of drive) are included in this section since drive is considered 
to be one of uhe more important variables leading to success. POTENTIAL also 
is included here since potential is a measure of future performance. AWARDS 
ANL rUNISHMENT, REPUTE, and ASSET TO THE NAVY represent acknowledgments of an 
individuals performance, either positive or negative acknowledgment. 

Each sentence of narrative text in the pilot study sample and the cross 
validation and generalization samples was read carefully and, where appropri- 
ate, divided into segments corresponding to the assignment of specific index 
terms. However, it is not enough to simply label a narrative statement with 
the most appropriate index term since the statement may have been a highly 
positive, quite positive, neutral, quite negative, or highly negative one* 
For xample, in order to differentiate between the ratee who plans superbly 
and he ratee who plans inadequately, a weighting scale was devised to be 
applied to each index term that is used (see Table 7). The weighting scale 
contains five numerical values ranging from 5 (the positive end of the scale) 
to 1 (the negative end of the scale). Under each numerical value in Table 7 
there are listed samples of adjectives or adverbs that may be used by the 
rater to describe a ratee 1 s performance. These lists of words provide clues 
to the indexer as to which numerical value to assign to an index term. As a 
simple example, if the rater commented that the ratee was highly cooperative, 
this statement would be indexed as COOPERATION and assigned a weight of 4 since 
highly is listed as an example under numeral 4 in Table 7. 

The weighting scale developed for this content analysis research bears a 
marked resemblance to the quality rating scale developed by Harrington in rat- 
ing narrative statements contained in letters of recommendation regarding can- 
didates for secondary level teaching positions. 11 Harrington's quality rating 
scale was based upon the proposition that the f avorableness of the modifying 
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excellent 



4 

_good 



TABLE 7 
WEIGHTING SCALE 

3 

1 average | 



2 

poor 



1 

poorest 



superlative 
best 



comparative 

better than 
most 



average 



comparative 

not as good 
as most 



superlative 
worst 



EXAMPLES 



above 


aDOVc dvcLdgc 




QcC 1 llllllg 


reproach 


better 


aptly 


quality 


u c y \j LiLL 


rriTTiPTi 21 n 1 P 

UUU1C11U d LS _1_ 


f* a na Vi 1 p 
d pa u j_ c 


r^pf 1 r» n orir 1 v 
L1CJL XV^XCU'w jr 






r* r\vn not*DTif 
L.UUip CLcllL 


UcLL XT lie 1 1 Lai 


Douna les s 


deep 


genera Liy 


fair 


cp L .LOIlci 1 


Ucj. J. 11 J. Lciy 


moaer a u e 


m uGcu or 


ext ra- 


o a c i 1 v 


odLlbldL" 


lllbUI 1 XClcIlL 


ni "V™ "1 Y"% o "V™ ^ T 

orcxnary 


cl I cC L IVc 


tory 


lack of 


cAtrciiicJ.y 


or f ■? /-> -J on f" 

CJ.J.J-L.J.CL1L 


surncient— 


lower cnan 


finest 






average 


£ lawless 


cm men l 


usually 


lowering or 


greatest 


exceeds 




negatively 


highest 


excels 




spotty 


ideal 


exemplary 




unfortunate 


little to be 


expeditious 




unwisely 


desired 


experienced 




weak in 


limitless 


expertise 




with the ex- 


maximum 


extensive 




ception of 


most 


favorable 






never 


great 






outstanding 


high/highly 






paramount 


immaculate 






perfect 
profound 


immensely 
impeccable 


4-good (Cont.) 




sterling 


impressive 


rare 




superb 


innate 


remarkable 




superior 


inspires 


sign if icantly 




surpassed by 


instills 


skillful 




none 


invaluable 


smoothly 




top/topnotch 


keen 


solid 




unimpeachable 


laudable 


strongly 




unique 


leading 


surpassed 




unlimited 


marked 


thorough 




unmatched 


meticulously 


tremendous 




utmost 


model 


truly 




without equal 


much 


unstinting 




without 


noteworthy 


valuable 




exception 


particularly 


vast 




100% 


rapidly 


very 





bottom 

least 

lowest 



NOTE: AWARDS AND PUNISHMENT is assigned a weight of either 5 or L 
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terms used by the writer of a recommendation is an index of his enthusiasm in 
recommending the candidate and, therefore, of the quality of the recommenda- 
tion. The rationale for the weighting scale used in this study was similar to 
Harrington's proposition, A long list of adjectives and adverbs found to 
occur in a large corpus of Evaluation Reports were classified into five de- 
grees of f avorableness/unf avorableness in modifying terms. There is a great 
deal of overlap between the adjectives and adverbs contained in Harrington } s 
quality rating scale and those contained in the weighting scale used in this 
research shown in Table 7. There also is a high degree of correspondence in 
the relative positioning of the adjectives and adverbs along the two scales. 
The existence of the fairly ancient Harrington monograph was not discovered 
until after the weighting scale used in this research had been devised and the 
research being reported had been completed. It is interesting that two re- 
search efforts conducted 30 years apart each independently developed a similar 
conceptual framework for performing a content analysis of narrative recommen- 
dations or evaluations. Harrington's procedure was identical to the one fol- 
lowed in the- research being reported here. The narrative text of each Evalua- 
tion Report was read, segmented into distinct statements, and each statement 
was then assigned one or more index terms from the set of 29 possible choices 
shown in Figure 2. Each term selected was also assigned a numerical weight 
from 1 to 5 depending upon the nature of the adjectives or adverbs used as 
modifiers in the statement. The following examples will make more explicit 
the indexing procedure that was followed. 

Example 1 . "BTC has an excellent working and practical knowledge of the 
PMS System/but has a tendency to be lax in the administrative phase of the 
system. 11 

This sentence was segmented into two parts. The first part was indexed 
as TECHNICAL SKILLS and assigned a weight of 5. The second part was indexed 
as MANAGEMENT FUNCTIONS and assigned a weight of 2. 

Example 2 . "Chief XX was relieved of his duties as the ship's Oil King 
after serving in the capacity for approximately two months./ He was removed 
from this billet because of his lack of professional knowledge/and technical 
know-how in the art of refueling." 

This portion of narrative text was divided into three segments for index- 
ing purposes. Segment 1 was indexed as AWARDS AND PUNISHMENT and assigned a 
weight of 1. Segnr nt 2 was indexed as PROFESSIONALISM and assigned a weight 
of 2. Segment 3 was indexed as TECHNICAL SKILLS and assigned a weight of 2. 

Example 3 * "He is able to direct the efforts of Line Personnel in an 
efficient and effective manner;/this is reflected in the ratee by a multiple 
of exceptional qualities." 

This sentence was segmented into two parts. The first part was indexed 
as LEADERSHIP AND DIRECTING and assigned a weight of 4. The second part was 
indexed as SKILLS AND ABILITIES and assigned a weight of 5. 



Example 4 . "His natural abilities/and responsible approach to recruit- 
ing/have enabled the ratee to outperform his contemporaries." 
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This sentence was segmented into three parts. The first part was index- 
ed as SKILLS AND ABILITIES and assigned a weight of 3. The second part was 
indexed as RELIABILITY AND DEPENDABILITY and assigned a weight of 3, The 
third part was indexed as PRODUCTIVITY AND ACHIEVEMENT and assigned a weight 
of 4. 

Figure 3 shows an example of the complete narrative text written in an 
evaluation section. The index terms that were selected by the indexer have 
been recorded above each segment of text and the indexing weights that were 
assigned appear directly after each term- Factual statements requiring no in- 
dexing were enclosed in brackets. The number of words in the narrative text 
were counted and recorded at the bottom of the text by the indexer. 

After all of the narrative text for either an evaluation section or a 
justification section of an Evaluation Report was indexed, the weights corre- 
sponding to each term were written onto the indexing form to the right of the 
appropriate index term (see Figure 4) . Thus there may have been two instances 
of mention of the ratee's INTELLECTUAL FUNCTIONING, the first mention given a 
weight of 3 and the second a weight of 4. To the right of INTELLECTUAL FUNC- 
TIONING on the indexing form for this ratee would be written the following 
string of weights: 3,4. Then to the far right on the indexing form under the 
column headed n Freq. n would be written "2", indicating that this index term 
had been used two times in indexing that particular section of narrative text. 

At the bottom of the indexing form there is a line labeled FREQUENCY 
COUNTS. After all of the weights assigned to the index terms selected for a 
section of narrative text (19R or 19S) had been entered on the indexing form, 
all of the 5 weights were counted and the sum was entered to the right of 5 on 
the FREQUENCY COUNTS line. The same procedure was followed for entering the 
frequency count of 4 weights, 3 weights, 2 weights, and 1 weights. The final 
step in completing the indexing form was to transfer the total number of words 
written at the bottom of the narrative text and to count the total number of 
index terms selected from the set of 29 possibilities. 

In order to increase the likelihood of consistent usage of the indexing 
vocabulary, a definition was written for each of the 29 index terms. Koontz 
and O'Donnell's Principles of Management* 7 was relied upon heavily in defining 
the management-oriented terms listed in Figure 2. Also contributing to the 
formulation of the definitions for the 29 index terms was the way that Navy 
evaluators actually referred to these concepts in narrative text. These defi- 
nitions were consulted frequently during the indexing process. Indexing of 
the pilot study sample and the cross validation and generalization samples was 
performed by one experienced indexer who also had conceptualized the content 
of the indexing vocabulary and had prepared the definitions of the 29 terms. 
As part of the concomitant study to ascertain the reliability of this content 
analysis methodology, a training manual was developed for use by the four re- 
liability indexers participating in the study. This training manual is in- 
cluded in its entirety in Appendix A and incorporates an alphabetical diction- 
ary of the 29 index terms. The dictionary definition for each tern is follow- 
ed by extensive examples of correct indexing usage of the term and the proper 
assignment of weights* 
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As Baxendale has so cogently articulated, u The core problem of content 
analysis concerns language in both its connotative and denotative aspects — 
a Pandora's box of semantic and psychological complexities and unknowns, 11 - 12 
Acknowledging these obstacles to a perfect representation of the content of a 
particular segment of narrative text, numerous examples of indexing usage have 
been included in the alphabetical dictionary of index terms, primarily as a 
means of illustrating the level of objectivity and consistency that can be 
achieved in this type of content analysis, but also to demonstrate the intrac- 
table problems that still reside in any effort to organize and objectify the 
domain of linguistic discourse. 
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Ratee is an intelligent and proficient Petty Officer, who performs his duties 

in an outstanding manner. His ability to plan, organize, coordinate and super- 

vise have been ably demonstrated by his performance as Recruit Company Command- 

er. Ratee is dependable, trustworthy, and exhibits mature judgment in dispos- 
er fttr/ fc- 

ing of problems which occur within his company. Ratee' s military appearance 

CA + a r 3 40 T*>P ¥ co**J> J 

and neatness of person and dress denote great pride. He is cheerful, highly 

motivated, and gets along exceptionally well with others. Ratee 's command of 

the English language; both orally and written is above average. Ratee is high- 
er * p (09 

ly recommended for E-8. / Ratee has been in Water Survival and Hygiene Division 

- for- v- 

only for a short period of time.] He has shown a great potential towards being 

a swimming instructor. Ratee is practicing on his own time to qualify for 
Senior Life Saver. 

T * /3.T 



Figure 3. Example of the Narrative Text for An Evaluation Section Show- 
ing the Indexing Decisions That Were Made. Factual Statements 
Requiring No Indexing Are Enclosed in Brackets. T = Total 
Number of Words in the Narrative Text. 
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ID No. IOOO 



Index Term 



Criterion Grou p LL*^*-*y^ Sectio n € v-txjC . (t^rt ) 



MANAGEMENT FUNCTIONS_ 
CONTROLLING 



LEADERSHIP AND DIRECTING_ 

ORGANIZAT ION 

PLANNING 



REPRESENTATION^ 
STAFFING 



USE OF COMMUNICATION, 
SKILLS AND ABILITIES 



COMMUNICATION 



CONDUCT, INTEGRITY, AND PRIDE_ 

COOPERATION 

ENDURANCE 



FLEXIBILITY 



GROOMING AND ATTIRE_ 
INITIATIVE 



INTELLECTUAL FUNCTIONING_ 
PROFESSIONALISM 



RELIABILITY AND DEPEND ABILITY_ 

RESOURCEFULNE S S 

RESPONSIVENESS 



TECHNICAL SKILLS 



PRODUCTIVITY AND ACHIEVEMENT^ 

AWARDS AND PUNISHMENT 

DRIVE 



SERVICE MOTIVATION^ 

POTENTIAL 

REPUTE 



FREQUENCY COUNTS 



ASSET TO THE NAVY_ 
; 5 2 ,4 



r 



Freq. 



TOTAL NUMBER OF WORDS / 3 S 



TOTAL NUMBER OF INDEX TERMS I 3 



Figure 4. The Indexing Form As It Was Filled Out to Record the Indexing De- 
cisions Made in the Example of Narrative Text Shown in Figure 3 




SECTION 4. STATISTICAL ANALYSIS AND RESULTS 

A set of 67 quantitative variables was derived from the indexing form 
used in the content analysis (see Table 8) . The first 29 variables reflect 
the simple frequency with which each index term was used to index a particular 
section of narrative text. Variable 30 is the sum of these 29 frequencies. 
Variables 31 through 59 represent the weighted frequency of each index term 
used to index a particular section of narrative text. For example, suppose 
that the index term CONTROLLING was used twice. The first time that it was 
used it was assigned a weight of 4; the second time that it was used it was 
assigned a weight of 3. The weighted frequency then for CONTROLLING would be 
4x1 + 3x1 = 7. The simple frequency for this same example would be 1 + 1 = 2. 
Variable 60 is similar to Variable 30 in that it is the sum of the 29 weighted 
frequencies. 

Variables 61 through 65 represent the frequency counts over the entire 
indexing form for all 5 weights, 4 weights, 3 weights, 2 weights, and 1 
weights. Variable 66 is the total number of words in the section of narrative 
text that was indexed. Variable 67 is the total number of index terms of the 
29 available that were used to index the section of narrative text. 

Profiles or vectors of these 67 values then were prepared for all of the 
Evaluation Reports contained in each sample. Separate profiles were compiled 
for the evaluation and justification sections of each Evaluation Report. If 
certain index terms were not used at all in indexing the evaluation section 
narrative or the justification section narrative, they were given a value of 
zero in the profile. This practice raised an important theoretical issue. Is 
it more damaging not to say anything about a ra tee's performance in a particu- 
lar area than to damn him with qualified praise? A statement such as the fol- 
lowing was assigned a weight of 2: M With more time and conscientious effort, 
he should realize a greater potential." This evaluation of the ratee's poten- 
tial seems more negative than not to have commented at all about his potential. 

As a result of these considerations, the weighting scale that had been 
used in the indexing of Variables 31 through 59 was transformed in order to 
place no comment between positive comments and negative comments. Table 9 
shows the conversion that was used. A constant cf 10 was added to the weight- 
ed frequency of Variables 31 through 59 in order to avoid the incidence of any 
negative input values in the subsequent statistical computations. 

All profiles were transformed to the new weighting scale and entered onto 
IBM coding forms in preparation for keypunching. The criterion data and occu- 
pational rating codes were known for the pilot study sample and were included 
on the coding forms. However, all of the coding forms for the cross valida- 
tion sample and the generalization sample were sent to the Navy Personnel Re- 
search and Development Center in San Diego where the criterion data and occu- 
pational rating codes were added to the coding forms and then returned to R-K 
Research and System Design for keypunching at UCLA. Card decks for each of 
the three samples were assembled in six parts: (1) Upper Criterion Group - 
Evaluation Section, (2) Middle Criterion Group - Evaluation Section, (3) Lower 
Criterion Group - Evaluation Section, (4) Upper Criterion Group - Justifica- 
tion Section, (5) Middle Criterion Group - Justification Section, and (6) 

ERLC 



TABLE 8 



DEFINITION OF THE 67 QUANTITATIVE VARIABLES 
DERIVED FROM THE INDEXING FORM 



Number of 

Variable Description of Variable 

1 Frequency of Mention of MANAGEMENT FUNCTIONS 

2 Frequency of Mention of CONTROLLING 

3 Frequency of Mention of LEADERSHIP AND DIRECTING 

4 Frequency of Mentron of ORGANIZATION 

5 Frequency of Mention of PLANNING 

6 Frequency of Mention of REPRESENTATION 

7 Frequency of Mention of STAFFING 

8 Frequency of Mention of USE OF COMMUNICATION 

9 Frequency of Mention of SKILLS AND ABILITIES 

10 Frequency of Mention of COMMUNICATION 

11 Frequency of Mention of CONDUCT, INTEGRITY, AND 

PRIDE 

12 Frequency of Mention of COOPERATION 

13 Frequency of Mention of ENDURANCE 

14 Frequency of Mention of FLEXIBILITY 

15 Frequency of Mention of GROOMING AND ATTIRE 

16 Frequency of Mention of INITIATIVE 

17 Frequency of Mention of INTELLECTUAL FUNCTIONING 

18 Frequency of Mention of PROFESSIONALISM 

19 Frequency of Mention of RELIABILITY AND DEPENDA- 

BILITY 

20 F. , luency of Mention of RESOURCEFULNESS 

21 Frequency of Mention of RESPONSIVENESS 

22 Frequency of Mention of TECHNICAL SKILLS 

23 Frequency of Mention of PRODUCTIVITY AND 

ACHIEVEMENT 

24 Frequency of Mention of .AWARDS AND PUNISHMENT ■ 

25 Frequency of Mention of DRIVE 

26 Frequency of Mention of SERVICE MOTIVATION 

27 Frequency of Mention of POTENTIAL 

28 Frequency of Mention of REPUTE 

29 Frequency of Mention of ASSET TO THE NAVY 

30 Sum of Variables 1 through 29 

31 Weighted Frequency of Mention of MANAGEMENT 

FUNCTIONS 

32 Weighted Frequency of Mention of CONTROLLING 

33 Weighted Frequency of Mention of LEADERSHIP 

AND DIRECTING 

34 Weighted Frequency of Mention of ORGANIZATION 

35 Weighted Frequency of Mention of PLANNING 

36 Weighted Frequency of Mention of REPRESENTATION 



(Continued) 
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TABLE 8 (CONT.) 

DEFINITION OF THE 67 QUANTITATIVE VARIABLES 
DERIVED FROM THE INDEXING FORM 



Number of 

Variable Description of Variable 



37 


Weighted 


Frequency 


of 


Mention of STAFFING 


38 


Weighted 


Frequency 


of 


Mention of USE OF 










COMMUNICATION 


39 


Weighted 


Frequency 


of 


Mention of SKILLS AND 










ABILITIES 


40 


Weighted 


Frequency 


of 


Mention of COMMUNICATION 


41 


Weighted 


Frequency 


of 


Mention of CONDUCT, INTE- 










GRITY, AND PRIDE 


42 


Weighted 


Frequency 


of 


Mention of COOPERATION 


43 


Weighted 


Frequency 


of 


Mention of ENDURANCE 


44 


Weighted 


Frequency 


of 


Mention of FLEXIBILITY 


45 


Weighted 


Frequency 


of 


Mention of GROOMING AND 










ATTIRE 


46 


Weighted 


Frequency 


of 


Mention of INITIATIVE 


47 


Weighted 


Frequency 


of 


Mention of INTELLECTUAL 










FUNCTIONING 


48 


Weighted 


Frequency 


of 


Mention of PROFESSIONALISM 


49 


Weighted 


Frequency 


of 


Mention of RELIABILITY AND 










DEPENDABILITY 


50 


Weighted 


Frequency 


of 


Mention of RESOURCEFULNESS 


51 


Weighted 


Frequency 


of 


Mention of RESPONSIVENESS 


52 


Weighted 


Frequency 


of 


Mention of TECHNICAL SKILLS 


53 


Weighted 


Frequency 


of 


Mention of PRODUCTIVITY AND 










ACHIEVEMENT 


54 


Weighted 


Frequency 


of 


Mention of AWARDS AND 










PUNISHMENT ' 


55 


Weighted 


Frequency 


of 


Mention of DRIVE 


56 


Weighted 


Frequency 


of 


Mention of SERVICE MOTIVA- 










TION 


57 


Weighted 


Frequency 


of 


Mention of POTENTIAL 


58 


Weighted 


Frequency 


of 


Mention of REPUTE 


59 


Weighted 


Frequency 


of 


Mention of ASSET TO THE 



NAVY 

60 Sum of Variables 31 through 59 

61 Total Number of 5 Weights 

62 Total Number of 4 Weights 

63 Total Number of 3 Weights 

64 Total Number of 2 Weights 

65 Total Number of 1 Weights 

66 Total Number of Words in Narrative Text 

67 Total Number of Index Terms Used 
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TABLE 9 



TRANSFORMATION OF WEIGHTING SCALE 



Original 
Weights 



Transformed 
Weights 



5 (Excellent) 
4 (Good) 
3 (Average) 



3 (Excellent) 

2 (Good) 

1 (Average) 

0 (No Comment) 

1 (Poor) 

2 (Poorest) 



2 (Poor) 
1 (Poorest) 
0 (No Comment) 



Lower Criterion Group - Justification Section, Duplicate card decks were sent 
to NPRDC. 

In order to better visualize the nature of the distributions of the 67 
variables for each of the three research samples, computer runs were made at 
the UCLA Health Sciences Computing Facility* using Program BMD01D of the li- 
brar\ of Biomedical Computer Programs, 13 Program BMD01D - Simple Data Descrip- 
tion computes arithmetic means, standard deviations, standard errors of means, 
maximum values, minimum values, ranges, and sample sizes for a set of input 
variables. The output from these computer runs for the cross validation sam- 
ple and the generalization sample is presented in Appendix B, The output from 
l:he computer run for the pilot study sample was included in an earlier techni- 
cal report, lif and since it is rather voluminous, it is not repeated in this re- 
port. 

Some interesting comparisons can be made from the descriptive statistics 
compiled in Appendix B. Tables 10 and 11 present the data on Variable 66, to- 
tal number of words contained in the narrative text of the evaluation and jus- 
tification sections of the Evaluation Report, Table 10 for the cross valida- 
tion sample and Table 11 for the generalization sample. The results for the 
evaluation sections of Tables 10 and 11 are different from those found in the 
pilot study 15 where the nearer that the criterion group was to the top of the 
distribution of T Scores on 1 9A-PERF0RMANCE OF DUTY, the longer the narrative 
text written to evaluate the performance of the ratee. In the pilot study 
sample approximately 89 words were written on the average to evaluate ratees 
in the Upner criterion group. This average dropped to approximately 76 words 
in the Middle criterion group and to approximately 67 words in the Lower cri- 
terion group. This decreasing trend is not apparent in Tables 10 and 11 for 
the evaluation section. Instead, in the cross validation sample the trend is 
just the opposite, with the average length of the narrative text of the evalua- 
tion section becoming longer as the criterion group moves farther away from 



Computing assistance was obtained from the Health Sciences Computing Facility, 
UCLA, sponsored by NIH Special Research Resources Grant RR-3. 
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TABLE 10 

DESCRIPTIVE STATISTICS ON VARIABLE 66: 
TOTAL NUMBER OF WORDS IN NARRATIVE TEXT 
FOR THE CROSS VALIDATION SAMPLE 



Evaluation 
Section 

Upper 

Middle 

Lower 



Mean 

78.3376 
86.5808 
87.1889 



S.D. 

49.0724 
42.8689 
46.2204 



Max. 

212 
214 
287 



Min. 

0 
0 
0 



Range 

212 
214 
287 



Justification 
Section 

Upper 

Middle 

Lower 



Mean 

201.4052 
146.5944 
17.6754 



S.D. 

146.3721 
124.8155 
27.2311 



Max. 

896 
820 
112 



Min. 

35 
17 
0 



Range 

861 
803 
112 



TABLE 11 

DESCRIPTIVE STATISTICS ON VARIABLE 66: 
TOTAL NUMBER OF WORDS IN NARRATIVE TEXT 
FOR THE GENERALIZATION SAMPLE 



Evaluation 
Section 

Upper 

Middle 

Lower 



Mean 

98.1754 
84.5132 
100.2835 



S.D. 

83.3072 
53.0552 
54.5660 



Max. 

530 
293 
353 



Min. 

0 
0 
0 



Range 

530 
293 
353 



Justification 
Section 

Upper 

Middle 

Lower 



Mean 

228.2968 
133.0268 
27.4051 



S.D. 

155.7997 
96.9929 
40.0231 



Max. 

881 
442 
172 



Min. 

17 
0 
0 



Range 

864 
442 
172 



ERIC 
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the top of the distribution of T Scores on 19A. In the generalization sample 
the average length of the narrative text of the evaluation section drops from 
the Upper criterion group to the Middle criterion group but then rises even 
higher for the Lower criterion group than for the Upper criterion group. These 
unexpected findings for the cross validation and generalization samples can be 
explained by the presence of a few atypically long evaluation sections in the 
Middle and Lower criterion groups that markedly influenced the mean length for 
these two criterion groups. This explanation is further corroborated by exam- 
ining the maximum total number of words contained in the narrative text of the 
evaluation section for the Lower criterion group as shown in Tables 10 and 11. 
A minimum of zero words resulted from evaluation sections not being written at 
all for a handful of cases in the various criterion groups. 

In the pilot study the trend of longer narrative text for criterion groups 
near the top of the distribution of T Scores on 19A-PERF0RMANCE OF DUTY pre- 
vailed also for the justification section, but the effect was even more pro- 
nounced than for the evaluation section. This same outcome resulted for the 
justification section in the cross validation and generalization samples, and 
can be explained partly by the requirement to write an individual justification 
in Section 19S for any mark given in the top 10, 5, or 1% columns of Section 
19. When a justification section was written, on the average it was longer 
than the evaluation section in the Upper and Middle criterion groups of all 
three samples. 

Tables 12 and 13 also extract data from Appendix B and present descriptive 
statistics for all three criterion groups in the cross validation and generali- 
zation samples on Variable 67, total number ^of index terms used of the 29 
available to index the evaluation and justification sections of the Evaluation 
Report. The results shown in Tables 12 and 13 are correlated to those present- 
ed in Tables 10 and 11 in that the number of index terms used is a function of 
length of the narrative text. As the evaluator uses more words to describe 
the ratee, he is more likely to comment on a wider variety of specific areas 
of the ratee' s performance. This tendency is mirrored in the number of dif- 
ferent index terms selected by the indexer to encompass the narrative content. 
It is interesting that on the average in the evaluation section of either the 
cross validation or the generalization sample only a half dozen substantive 
areas of the ratee* s performance were described of the 29 possibilities. This 
was also the finding in the pilot study. 16 Even in the justification section 
where longer expositions were written, on the average only 11 or 12 of the 29 
content areas were mentioned in the Upper criterion group (nine areas in the 
pilot study Upper criterion group 16 ). This finding, consistent across all 
three samples, suggests that the evaluators may be victims of habitual ways of 
formulating and phrasing the narrative sections of the Evaluation Report. Of 
the richness of information that could be used to describe the performance of 
ratees in the Upper criterion group, on the average only a partial representa- 
tion is utilized. All of the 29 index terms were used at one time or another 
to index the evaluation sections or the justification sections of the three 
criterion groups in all three samples. Therefore, one can conclude that al- 
though only a partial representation of the 29 substantive areas may be utiliz- 
ed to describe a particular ratee, over a sample as large as any one criterion 
group (N=74 or 75), all 29 areas of performance do get mentioned at one time 
or another. 
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TABLE 12 

DESCRIPTIVE STATISTICS ON VARIABLE 67 ; 
TOTAL NUMBER OF INDEX TERMS USED 
FOR THE CROSS VALIDATION SAMPLE 



Evaluation 
Section 

Upper 

Middle 

Lower 



Mean 

5.6081 
7.1621 
6.5135 



S.D. 

3.3508 
3.2011 
2.9759 



Max. 

14 
15 
16 



Min. 

0 
0 
0 



Range 

14 
15 
16 



Justification 
Section 

Upper 

Middle 

Lower 



Mean 

11.6081 
9.1621 
1.6081 



S.D. 

4.8224 
4.3132 
2.4372 



Max. 

27 
19 
9 



Min. 

4 
1 
0 



Range 

23 
18 

9 



TABLE 13 

DESCRIPTIVE STATISTICS ON VARIABLE 67: 
TOTAL NUMBER OF INDEX TERMS USED 
FOR THE GENERALIZATION SAMPLE 



Evaluation 

Section Mean S.D. Max. Min. Range 

Upper 6.3378 4.7637 22 0 22 

Middle 5.8648 3.3365 14 0 14 

Lower 6.3648 2.8020 14 0 14 

Justification 

Section Mean S.D. Max. . Min. Range 

Upper 12.4730 4.6852 22 1 21 

Middle 8.4730 4.2819 18 0 18 

Lower 1.8648 2.5660 10 0 10 



9 
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From the duplicate deck of punched cards for the cross validation and the 
generalization samples that was sent to NPRDC, correlation matrices were com- 
puted at the computing facility of the Naval Electronics Laboratory Center in 
San Diego. Correlations among all 67 variables for the evaluation section and 
for the justification section were computed for each sample as well as the ma- 
trix correlating the evaluation section with the justification section on all 
67 variables. These results are displayed in Appendix C. The reader is refer- 
red to Table 8 for a definition of each of the 67 variables. 

The overall picture that emerges from a scrutiny of these six matrices is 
that there are very few high correlations except for the correlations between 
Variables 1 through 30 and Variables 31 through 60 which are shown in italic 
type. These two sets of variables are the same except for the method of 
weighting that was used. The other variables that showed a consistently high 
correlation in the evaluation section and also in the justification section 
for both samples were Variable 30 (Sum of Variables 1 through 29) and Variable 
60 (Sum of Variables 31 through 59) with Variable 61 (Total Number of 5 [New 
3] Weights), Variable 62 (Total Number of 4 [New 2] Weights), Variable 63 (To- 
tal Number of 3 [New 1] Weights), Variable 66 (Total Number of Words in the 
Narrative Text), and Variable 67 (Total Number of Index Terms Used). Variable 
67 (Total Number of Index Terms Used) is a focal variable, correlating highly 
with Variable 30 (Sum of Variables 1 through 29) , Variable 60 (Sum of Variables 
31 through 59), Variable 61 (Total Number of 5 [New 3] Weights), Variable 62 
(Total Number of 4 [New 2] Weights), Variable 63 (Total Number of 3 [New 1] 
Weights), and Variable 66 (Total Number of Words in the Narrative Text). Cor- 
relations among the variables having to do with the 29 index terms per se were 
notably low, indicating that these 29 dimensions are relatively orthogonal and 
represent independent aspects of managerial performance. In the correlation 
matrix for the evaluation section versus the" justification section, no high 
correlations were evident in either the cross validation sample or the gener- 
alization sample, demonstrating that these two narrative sections of Evalua- 
tion Report Form NAVPERS 1616/8 are quite independent and should be treated 
separately. An early decision in this research project was to treat these two 
narrative sections separately; in retrospect this determination appears to 
have been a wise decision. 

Since the system that was used for weighting Variables 31 through 59 has 
been highlighted by the correlational analysis as an important aspect of the 
content analysis methodology, it is of interest to see the distribution of in- 
dex weights used for each of the three criterion groups in the cross valida- 
tion and generalization samples. These results are presented in Tables 14 
through 17. Tables 14 and 15 show the distribution for the cross validation 
sample, first for the evaluation section and then for the justification sec- 
tion. Comparable distributions for the generalization sample are shown in 
Tables 16 and 17. A chi square test of common distribution was calculated for 
all pairwise criterion group comparisons in Tables 14 through 17 in order to 
test the hypothesis that the distributions of index weights used for each pair 
of groups were drawn from the same populatd This hypothesis was rejected 

beyond the .001 level of probability for f: f the six comparisons made for 
the cross validation sample (see Table 18). i/or the comparison of the Upper 
versus Middle criterion groups on the justification section, the hypothesis 
was rejected beyond the .01 level of probability. These results replicate the 
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TABLE 14 

DISTRIBUTION OF TOTAL NUMBER OF 5 WEIGHTS THROUGH 1 WEIGHTS 
USED IN INDEXING THE EVALUATION SECTION (19R) 
OF THE CROSS VALIDATION SAMPLE FOR ALL THREE CRITERION GROUPS 







Criterion Group 


Weights 




Upper 


Middle 


Lower 


5 (New 3) Excellent 




203 


157 


34 


4 (New 2) Good 




270 


324 


314 


3 (New 1) Average 




144 


261 


234 


2 (New -1) Poor 




0 


14 


71 


1 (New -2) Poorest 




0 


0 


1 



TABLE 15 

DISTRIBUTION OF TOTAL NUMBER OF 5 WEIGHTS THROUGH 1 WEIGHTS 
USED IN INDEXING THE JUSTIFICATION SECTION (19S) 
OF THE CROSS VALIDATION SAMPLE FOR ALL THREE CRITERION GROUPS 



Weights 


Criterion Group 


Upper 


Middle 


Lower 


5 


(New 3) Excellent 


508 




265 


48 


4 


(New 2) Good 


738 




486 


71 


3 


(New 1) Average 


528 




408 


38 


2 


(New -1) Poor 


0 




2 


2 


1 


(New -2) Poorest 


0 




0 


1 



TABLE 16 



DISTRIBUTION OF TOTAL NUMBER OF 5 WEIGHTS THROUGH 1 WEIGHTS 

USED IN INDEXING THE EVALUATION SECTION (19R) 
OF THE GENERALIZATION SAMPLE FOR ALL THREE CRITERION GROUPS 



Weights 


Criterion Group 


Upper 


Middle 


Lower 


5 


(New 3) Excellent 


220 


137 




115 


. A 


(New 2) Good 


327 


287 




317 


3 


(New 1) Average 


199 


205 




211 


2 


(New -1) Poor 


1 


. 17 




54 


1 


(New -2) Poorest 


0 


0 




1 



TABLE 17 

DISTRIBUTION OF TOTAL NUMBER OF 5 WEIGHTS THROUGH 1 WEIGHTS 

USED IN INDEXING THE JUSTIFICATION SECTION (19S) 
OF THE GENERALIZATION SAMPLE FOR ALL THREE CRITERION GROUPS 



Weights 


Criterion Group 


Upper 


Middle 


Lower 


5 


(New 3) Excellent 


451 


231 


46 


A 


(New 2) Good 


783 


kkk 


105 


3 


(New 1) Average 


579 


385 


51 


2 


(New -1) Poor 


5 


3 


3 


1 


(New -2) Poorest 


0 


0 


0 



TABLE 18 



RESULTS OF THE CHI SQUARE TESTS OF COMMON DISTRIBUTION 
OF INDEX WEIGHTS USED FOR EACH PAIR OF CRITERION GROUPS 
IN THE CROSS VALIDATION SAMPLE 



Criterion Group 
Comparison 

Evaluation Section 



Value of 
Chi Square 



Degrees of 
Freedom 



Probability 
Level 



Upper vs. Middle 


44.9757 


4 


P < 


.001 


Middle vs. Lower 


61.1906 


4 


P < 


.001 


Upper vs. Lower 


140.9668 


4 


P < 


.001 



Justification Section 



Upper vs. Middle 
Middle vs. Lower 
Upper vs. Lower 



18.4300 
21.3217 
35.4408 



4 
4 
4 



,001 < P < .01 
P < ,001 
P < .001 



TABLE 19 

RESULTS OF THE CHI SQT ARE TESTS OF COMMON DISTRIBUTION 
OF INDEX WEIGHTS USED FOR EACH PAIR OF CRITERION GROUPS 
IN THE GENERALIZATION SAMPLE 



Criterion Group 
Comparison 

Evaluation Section 



Value of 
Chi Square 



Degrees of 
Freedom 



Probability 
Level 



Upper vs. Middle 


29.0438 


4 


P < 


.001 


Middle vs, Lower 


21.7997 


4 


P < 


.001 


Upper vs. Lower 


83.9246 


4 


P < 


.001 



Justification Section 



Upper vs. Middle 
Middle vs. Lower 
Upper vs. Lower 



6.7782 
15.0797 
12.6617 



4 
4 
4 



P > .05 
.001 < P < .01 
.01 < P < .05 



Note: x with 4 degrees of freedom = 18.465. 
•001 
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findings in the earlier pilot study. / For the generalization sample the hy- 
pothesis was rejected beyond the .001 level of probability for all three com- 
parisons made for the evaluation section (see Table 19). However, in the jus- 
tification section the difference between the distributions of weights for the 
Upper versus 'Middle criterion groups was not statistically significant. The 
other two comparisons (i.e., Middle vs. Lower and Upper vs. Lower) were sta- 
tistically significant. The conclusion that can be drawn from Tables 14 
through 19 is that the higher the criterion group, the more excellent and good 
attributes that are mentioned by the evaluator. Thei lower the criterion group 
the more average, poor, and poorest incidents of behavior that are cited by th 
evaluator. 

In the pilot study the Mann-Whitney U test was used to test the null hy- 
pothesis that there were no differences between each pair of criterion groups 
on each of the 67 quantitative variables derived from the indexing form. The 
nonparametric Mann-Whitney U test was used in preference to the parametric t 
test of difference between two means because very few of the 67 variables were 
normally distributed and the t test assumes normality and common variance. 
However, since the t test is very robust, it was decided subsequently to also 
compute t tests of mean difference for the pilot study sample using Program 
BMDP3D in the library of Biomedical Computer Programs 13 at the UCLA Health 
Sciences Computing Facility. The results of these computer runs agreed sub- 
stantially with the results obtained using the Mann-Whitney U test, primarily 
because the sample size of the various criterion groups in the pilot study sam 
pie was equal and large (N=75) . The probability level of these two statisti- 
cal tests differed occasionally and only when the result fell near the bounda- 
ry of the various bench marks of statistical significance (e.g., the t test 
might be significant between the .01 and .001 probability levels and the Mann- 
Whitney U test might be significant between the .05 and .01 probability lev- 
els). 

As a consequence of the pilot study experience, both the Mann-Whitney U 
test and the t te.st of mean difference were computed on each of the 67 varia- 
bles for the cross validation and generalization samples. These computations 
were made for each pair of criterion groups in both the evaluation and the jus 
tification sections. The Mann-Whitney U tests were performed at the computing 
facility of the Naval Electronics Laboratory Center in San Diego under the su- 
pervision of NPRDC. The t tests were performed at the UCLA Health Sciences 
Computing Facility. The results of these computer runs are presented in Appen 
dix D. This appendix reports the number and name of the variable, the value 
of U, its associated absolute z value (approximately a normal deviate), and 
the corresponding probability level for a 2-tailed test. The convention that 
was followed in the Mann-Whitney U tests was to enter data into the computer 
program first for the lower criterion group in the comparison. For those z 
values that achieved statistical significance, the direction of the difference 
is that the higher criterion group evidenced a greater frequency or greater 
weighted frequency of the variable. The Mann-Whitney U test probability level 
in Appendix D is followed by the value of t and its probability level for a 2- 
tailed test. Thus the reader can compare the results of the two statistical 
tests for each variable. 

Those variables resulting in a statistically significant difference in 
each pairwise criterion group comparison have been extracted from Appendix D 
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and summarized in four tables Tables 20 tlirough 23, Only those variables 

showing a significant difference on both statistical tests have been extracted. 
Table 20 lists the statistically significant variables for the evaluation sec- 
tion of the cross validation sample. The easiest discrimination to be made 
should be between the Lower and Upper criterion groups since they are the most 
widely separated on the criterion variable 19A-PERF0RMANCE OF DUTY, Table 20 
reveals that eight of the 67 variables showed a statistically significant dif- 
ference between the Lower and Upper criterion groups. Seven of these eight 
variables also showed a statistically significant difference between the Lower 
and Upper criterion groups in the pilot study sample, 18 the exception being 
Total Number of 3 (New 1) Weights (Average) , In evaluating the performance of 
outstanding chief petty officers as compared to average CPO's, the significant 
differences reside in the superiority of the top criterion group with respect 
to professionalism, managerial know-how, more awards and fewer punishments, 
more superlative attributes, and fewer qualified statements reflecting fair or 
poor performance. 

By comparison Table 20 shows that there were eleven statistically signifi 
cant differences between the Lower and Middle criterion groups of the cross 
validation sample on the evaluation section. Only one of these eleven varia- 
bles showed a statistically significant difference between the Lower and Mid- 
dle criterion groups in the pilot study sample Total Number of 2 (New -1) 

Weights (Poor). 18 In the cross validation sample the areas of performance 
that differentiated average CPO's from those who are marked as somewhat supe- 
rior in performance of duty were professionalism, cooperation, responsiveness, 
resourcefulness, and productivity and achievement, Ratees in the middle cri- 
terion group were described with more superlatives and with fewer qualified 
statements of fair or poor performance. 

The most difficult discrimination to be made is that between the Middle 
and Upper criterion groups. Table 20 shows that despite this difficulty, 
eight statistically significant differences resulted from the comparison on 
the evaluation section. Only two of these eight variables showed a statisti- 
cally significant difference between the Middle and Upper criterion groups in 

the pilot study sample f of COOPERATION and wf of COOPERATION. 18 In the 

cross validation sample the characteristics that differentiated outstanding 
CP0 T s from their slightly less qualified colleagues were cooperation, grooming 
and attire, resourcefulness, and productivity and achievement. A wider range 
of index terms was used to index the Evaluation Reports of outstanding CP0 s 
which also resulted in more 3 (New 1) weights being assigned to this criterion 
group . 

Moving now to a consideration of the justification section for the cross 
validation sample (Table 21), all but five of the 67 variables showed a statis- 
tically significant difference between the Lower and Upper criterion groups 
and all but eight of the 67 variables showed a statistically significant dif- 
ference between the Lower and Middle criterion groups. This plethora of sig- 
nificant results is an artifact occasioned by the requirement to write an in- 
dividual justification for any mark in the top 10, 5, or 1% columns of Section 
19, Despite this built-in bias, five of the 67 variables were not statistical- 
ly significant for the Lower versus Upper criterion group comparison: f of 
FLEXIBILITY, wf of FLEXIBILITY, wf of ORGANIZATION, Total Number of 2 (New -1) 
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Weights (Poor), and Total Number of 1 (New -2) Weights (Poorest), No 2 or 1 
weights were used in indexing the Upper criterion group and only two 2 weights 
and one 1 weight were used in indexing the Lower criterion group (see Table 
15). Eight of the 67 variables were n f statistically significant for the Low- 
er versus Middle criterion group comp^ \: f of ENDURANCE, wf of ENDURANCE, 
f of FLEXIBILITY, wf of FLEXIBILITY, f AWARDS AND PUNISHMENT, wf of INTEL- 
LECTUAL FUNCTIONING, Total Number of 2 (New -1) Weights (Poor), and Total Num- 
ber of 1 (New -2) Weights (Poorest). Only two 2 weights and no 1 weights were 
used in indexing the Middle criterion group compared to two 2 weights and one 
1 weight for the Lower criterion group (see Table 15). 

The only really cogent comparison for the justification section is be- 
tween the Upper and the Middle criterion groups because both of these groups 
required justification comments. Table 21 shows that 21 of the 67 variables 
evidenced a statistically significant difference between these two criterion 
groups, eight of which also showed a statistically significant difference be- 
tween the Upper and Middle criterion groups in the pilot study sample wf of 

SKILLS AND ABILITIES, f of PRODUCTIVITY AND ACHIEVEMENT, wf of PRODUCTIVITY 
AND ACHIEVEMENT, Sum of Variables 1 through 29, Sum of Variables 31 through 59, 
Total Number of 5 (New 3) Weights, Total Number of Words in Text, and Total 
Number of Index Terms Used. 19 When the evaluator is required to justify his 
marks in evaluating a ratee, he apparently calls out certain areas of perform- 
ance that distinguish the ratee in the Upper criterion group from his slightly 
less qualified colleague in the Middle criterion group. In the cross valida- 
tion sample skills and abilities as well as productivity and achievement were 
the differentiating areas of performance. The ratee in the Upper criterion 
group also had bestowed upon nim more superlative adjectives and adverbs. All 
three criterion group comparisons on the justification section of the Evalua- 
tion Report showed significant differences on both Total Number of Words in 
Text and Total Number of Index Terms Used. 

Table 22 l_sts the statistically significant variables for the evaluation 
section of the generalization sample. Again, the easiest discrimination to be 
made should be between the Lower .and Upper criterion ^groups since they are the 
most widely separated on the criterion variable 19A-PERF0RMANCE OF DUTY. Only 
four of the 67 variables showed a statistically significant difference between 

the Lower and Upper criterion groups f of AWARDS AND PUNISHMENT, wf of AWARDS 

AND PUNISHMENT, Total Number of 5 (New 3) Weights (Excellent), and Total Number 
of 2 (New -1) Weights (Poor). All four of these variables also showed a sta- 
tistically significant difference between the Lower and Upper criterion groups 
on the evaluation section of the cross validation sample. 

By comparison Table 22 shows that there were six statistically significant 
differences between the Lower and Middle criterion groups of the generalization 
sample on the evaluation section. Only one of these six variables also showed 
a statistically significant difference between the Lower and Middle criterion 

groups on the evaluation section of the cross validation sample Total Number 

of 2 (New -1) Weights (Poor). Four statistically significant differences re- 
sulted from the comparison between the Middle and Upper criterion groups on 
the evaluation section of the generalization sample f of POTENTIAL, wf of PO- 
TENTIAL, Total Number of 5 (New 3) Weights (Excellent), and Total Number of 2 
(New -1) Weights (Poor) . None of these four variables overlapped with those 
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that were statistically significant for the Middle versus Upper criterion 
group comparison on the evaluation section of the cross validation sample. 
These results may indicate that the findings in the cross validation sample 
may not map particularly well onto the findings in the generalization sample, 
probably because of the differences in the occupational ratings represented in 
these two samples. This issue will be dealt with more thoroughly later in this 
section when the .results of the stepwise discriminant analyses are discussed. 

Moving now to a consideration of the justification section for the gener- 
alization sample (Table 23), all but nine of the 67 variables showed a statis- 
tically significant difference between the Lower and Upper criterion groups. 
Four of these nine variables also showed a lack of statistical significance 
between the Lower and Upper criterion groups on the justification section of 
the cross validation sample f of FLEXIBILITY, wf of FLEXIBILITY, Total Num- 
ber of 2 (New -1) Weights (Poor), and Total Number of 1 (New -2) Weights 
(Poorest). All but 15 of the 67 variables showed a statistically significant 
difference between the Lower and Middle criterion groups on the justification 
section of the generalization sample (Table 23). Five of these 15 variables 
also showed a lack of statistical significance between the Lower and Middle 

criterion groups on the justification section of the cross validation sample 

f of FLEXIBILITY, wf of FLEXIBILITY, f of AWARDS AND PUNISHMENT, Total Number 
of 2 (New -1) Weights (Poor), and Total Number of 1 (New -2) Weights (Poorest). 

As pointed out earlier, the only really cogent comparison for the justifi- 
cation section is between the Upper and the Middle criterion groups because 
both of these groups required justification comments. Table 23 shows that 38 
of the 67 variables evidenced a statistically significant difference between 
these two criterion groups in the generalization sample, 17 of which also 
showed a statistically significant difference between the Upper and Middle cri- 
terion groups in the cross validation sample. This finding suggests that there 
is partial overlap between these two samples. The nonoverlapping areas may be 
attributed to a difference in the nature of the occupational ratings represent- 
ed in the cross validation and the generalization samples and/or to unrelia- 
bility in the indexing procedure, although the results of the reliability study 
presented in Section 5 suggest that differences among the four occupational 
ratings provide a more reasonable explanation for these^ results than unrelia- 
bility. Both of these issues will be discussed more fully later in this re- 
port. 

In every pairwise criterion group comparison shown in Tables 20 through 
23, one or more of the five variables involving Total Number of 5 (New 3), 4 
(New 2), 3 (New 1), 2 (New -1), or 1 (New -2) Weights proved to be signifi- 
cantly differentiating. This finding supports the results of the correla- 
tional analysis presented in Appendix C in which the system used for weighting 
Variables 31 through 59 was highlighted as an important aspect of the content 
analysis methodology. 

Thus far in this report the 67 quantitative variables derived from the 
indexing form have been considered individually as potential discriminators 
among the three criterion groups in the evaluation and justification sections 
of the various samples. However, it is of considerable interest to learn how i 
well this set of variables, used in optimal combination, can classify each of 
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the research samples into correct criterion group* Therefore, the three sam- 
ples were also analyzed by Program BMD07M in the library of Biomedical Comput- 
er Programs 13 at the UCLA Health Sciences Computing Facility* This program 
performs a multiple discriminant analysis in a stepwise manner* At each step 
one variable is entered into the set of discriminating variables or a variable 
is deleted if its F value becomes too low* At the option of the user, a clas- 
sification matrix is computed and printed after those steps specified by the 
user* This option permits the user to determine if the classification process 
tends to converge to perfect classification or to maximize at some midway step 
and then disintegrate as more variables are added to the discriminant function* 

A special feature of Program BMD07M allows new cases to be classified by 
the discriminant functions generated on the original sample* This feature was 
used to perform two cross validation studies of the AT T s and BT's combined and 
also of the AT's and BT's considered separately* In the first study, the cross 
validation sample and the pilot study sample were both classified using the 
cross validation sample discriminant functions. In the second study, the pilot 
study sample and the cross validation sample were both classified using the pi- 
lot study sample discriminant functions* Figure 5 portrays the accuracy of 
classifying the cross validation sample and the pilot study sample into correct 
criterion group using the cross validation sample discriminant functions for 
the evaluation section (19R) • For the first three steps of the discriminant 
analysis the curves for the two samples are very close* Then, as the discrimi- 
nant analysis progresses step by step, the two" curves begin to separate with 
the accuracy of classifying the pilot study sample being considerably less than 
the accuracy of classifying the cross val" : d.:tion sample* It was expected, of 
course, that classification of the pilot study sample using the cross valida- 
tion sample discriminant functions would be less accurate than classification 
of the cross validation sample itself. Nonetheless, the pilot study sample 
curve tracks the behavior of the cross validation sample curve quite faithfully 
and even drops off precipitously at Step 64 as classification of the cross 
validation sample disintegrates. Figure 6 portrays the accuracy of classifying 
the cross validation sample and the pilot study sample into correct criterion 
group using the cross validation sample discriminant functions for the justi- 
fication section (19S) * As in Figure 5 the curves for the two samples remain 
very close for the first five steps after which they begin to diverge,' Again, 
less accuracy was achieved in classifying the pilot study sample using the 
cross validation sample discriminant functions than in classifying the cross 
validation sample itself. However, it should be noted that for both samples 
better classification accuracy was achieved on the justification section than 
on the evaluation section. This is more clearly shown in Table 24. 

Table 24 displays the best classification that was achieved on the evalua- 
tion section and the justification section of the cross validation sample and 
the pilot study sample using the cross validation sample discriminant func- 
tions. The underlined diagonal elements of the four classification matrices 
portrayed in Table 24 represent agreement between the statistical classifica- 
tion into criterion group and actual criterion group membership; the off- 
diagonal elements represent disagreement* The total number of statistical 
classifications matching actual classification is obtained by summing the di- 
agonal elements of each matrix, shown in Table 24 as the underlined diagonal 
sura. The step in the discriminant analysis at which this best classification 
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TABLE 24 

BEST CLASSIFICATION INTO THE THREE CRITERION GROUPS 
USING THE CROSS VALIDATION SAMPLE DISCRIMINANT FUNCTIONS 



EVALUATION SECTION - 19R 



CROSS VALIDATION SAMPLE (N=222) 



Step 64 



O iH 
•H 



4J 



•H ,Q 



O 



UPPER 



MIDDLE 



LOWER 



Classification by 
Discriminant Analysis 

UPPER MIDDLE LOWER 



63 5 
10 52 
9 11 



6 
12 
54 



< o 



Diagonal Sum = 169 



PILOT STUDY SAMPLE (N=224) 



Step 3 



O «H 

'jj w UPPER 
Tl 11 MIDDLE 

CJ 0) 

2 £< LOWER 

4J O 

q vi 



Classification by 
Discriminant Analysis 



UPPER 
54 
35 
23 



MIDDLE 
19 
23 
16 



Diagonal Sum 



LOWER 
2 
16 
36 
■ 113 



JUSTIFICATION SECTION - 19S 



CROSS VALIDATION SAMPLE (N=222) 



PILOT STUDY SAMPLE (N=224) 



Step 40 



O *H 

•h ,n 

M CO 
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4J Q) 

•H ,0 
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2 g< LOWER 

4J O 
U M 
< O 



UPPER 



MIDDLE 



Classification by 
Discriminant Analysis 

UPPER MIDDLE LOWER 



58 15 
8 61 
0 4 



1 
5 
70 



Step 4 



Diagonal Sum = 189 
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UPPER 

MIDDLE 

LOWER 



Classification by 
Discriminant Analysis 

UPPER MIDDLE LOWER 



43 
15 
0 



21 
45 
5 



11 
14 
70 



Diagonal Sum - 158 
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was achieved is also shown in Table 24 and corresponds to the maximum point on 
the curves shown in Figures 5 and 6. Of the 222 cases in the cross validation 
sample, 189 (85%) were classified correctly at Step 40 in the uiscriminant 
analysis of the justification section of the cross validation sample. Of the 
224 cases in the pilot study sample, 158 (71%) were classified correctly on 
the justification section with the cross validation sample discriminant func- 
tions, this maximum classification accuracy occurring at the fourth step. It 
should be pointed out that this presentation of the discriminant analysis re- 
sults? assumes that the criterion of actual group membership is perfect where 
in fact the possibility does exist that some of the members of the sample were 
given inflated marks on 19A-PERF0RMANCE OF DUTY, and consequently, were as- 
signed to an incorrect criterion group. Also, it is possible that narrative 
comments and evaluation marks may tap different aspects of performance. 

Figures 7 and 8 portray the accuracy of classifying the pilot study sam- 
ple and the cross validation sample into correct criterion group using the pi- 
lot study sample discriminant functions. Figure 7 shows these results for the 
evaluation section (19R) and Figure 8 shows them for the justification section 
(19S). As in the reverse cross validation shown in Figures 5 and 6, the two 
curves remain close together for the first five steps or so and then they di- 
verge with the pilot study sample being classified irore accurately on its owu 
discriminant functions than the cross validation sample. Once again better 
classification accuracy was achieved for both samples on the justification 
section. This is shown more clearly in Table 25. Of the 224 cases in the pi- 
lot study sample, 200 (89%) were classified correctly at Step 42 in the dis- 
criminant analysis of the justification section of the pilot study sample. Of 
the 222 cases in the cross validation sample, 1^9 (67%) were classified cor- 
rectly on the justification section with the pilot study sample discriminant 
functions, this maximum classification accuracy occurring at the third step. 

It was hypothesized that better classification would be achieved in using 
Program BMD07M if the two occupational ratings represented in each of the 
three research samples were analyzed separately. The skills needed to achieve 
superior performance may be quite different for Aviation Electronic^ Techni- 
cians than for Boilermen. Therefore, two cross validation studies also were 
conducted for the AT's and for the BT's. Figures 9 and 10 portray the accura- 
cy of classifying the cross validation AT T s and the pilot study AT r s into cor- 
rect criterion group using the cross validation AT f s discriminant functions. 
Figure 9 shows these results for the evaluation section (19R) and Figure 10 
shows them for the justification section (19S) . In both of these figures the 
two curves remain close together in the earlier steps and then diverge with 
the cross validation AT's being classified more accurately on their own dis- 
criminant functions than the pilot study AT's. As in the total cross valida- 
tion and pilot study samples, better classification accuracy was achieved for 
both samples on the justification section. Table 26 shows this more clearly. 
Of the 138 cross validation AT's, 129 (93%) were classified correctly at Step 
46 in the discriminant analysis of the justification section for the cross 
validation AT's. Of the 144 pilot study AT T s , 96 (67%) were classified cor- 
rectly on the justification section with the cross validation AT's discrimi- 
nant functions, this maximum classification accuracy occurring on the first 
step. 
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TABLE 25 

BEST CLASSIFICATION INTO THE THREE CRITERION GROUPS 
USING THE PILOT STUDY SAMPLE DISCRIMINANT FUNCTIONS 



EVALUATION SECTION - 19R 



PILOT STUDY SAMPLE (N=224) 



CROSS VALIDATION SAMPLE (N=222) 



Step 42 



O «H 

w UPPER 

4J <D 
•H £ 

a a) 

H 

■u O 

u u 

< o 



MIDDLE 



LOWER 



Classification by 
Discriminant Analysis 

UPPER MIDDLE LOWER 



49 
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44 



11 



15 



57 



Diagonal Sum - 150 



Step 4 

O •H 

U w UPPER 
a) n 
u (D 

U 'I MIDDLE 
S 

rt cx LOWER 

4-) O 

a n 
< o 



Classification by 
Discriminant Analyses 

UPPER MIDDLE LOWER 

34 25 15 

24 26 24 

8 14 52_ 

Diagonal Sum = 112 



JUSTIFICATION SECTION - 19S 



PILOT STUDY SAMPLE (N=224) 



CROSS VALIDATION SAMPLE (N=222) 



Step 42 
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Classification by 
Discriminant Analysis 

UPPER MIDDLE LOWER 
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Diagonal Sum = 200 
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TABLE 26 

BEST CLASSIFICATION INTO THE THREE CRITERION GROUPS ' 
USING THE CROSS VALIDATION AT T s DISCRIMINANT FUNCTIONS 



EVALUATION SECTION - 19F: 



CROSS VALIDATION AT T s (N=138) 



PILOT STUDY AT's (N=144) 



Step 48 



pi cx 

O *H 

u co 

U CD 

•H rO 

iH 

4J O 

O M 



UPPER 

MIDDLE 

LOWER 



Classification by 
Discriminant Analysis 



UPPER 
38 
7 
7 



MIDDLE 
3 
34 
4 



LOWER 
4 
3 
38 



Step 2 



Diagonal Sum = 110 



pi fx 

O tH 
•H 

M CO 

cu u 

4j aj 

•H ;Q 

U 0) 



§ §« LOWER 

4J O 

a m 



UPPER 



MIDDLE 



Classification by 
Discriminant Analysis 

UPPER MIDDLE LOWER 

26 10 13 

17 1 21 

10 8 38 

Diagonal Sum = 65 



JUSTIFICATION SECTION - 19S 



ERLC 



CROSS VALIDATION AT T s (N-138) 



PILOT STUDY AT's (N=144) 



Step 46 



pi cx 

O -H 

'H & 

U <0 

0) H 

4J CD 

•H ,Q 

M 0 

o| 

4J O 

o u 



UPPER 

MIDDLE 

LOWER 



Classification by 
Discriminant Analysis 



UPPER MIDDLE 

41 3 

3 40 

0 1 



LOWER 
1 
1 
48 

Diagonal Sum = 129 



Step 1 



O *rj 

M CO 

CD U 

4J d) 

u cu 

tH 

4J O 

o u 



Classification by 
Discriminant Analysis 





UPPER 


MIDDLE 


LOWER 


UPPER 


21 


16 


12 


MIDDLE 


11 


20 


8 


LOWER 


0 


1 


55 



Diagonal Sum = 96_ 



53 



The results of the reverse cross validation for the pilot study AT's and 
the cross validation AT T s are shown in Figures 11 and 12. Figure 11 portrays 
the accuracy of classifying the pilot study AT T s and the cross validation AT T s 
into correct criterion group on the evaluation section using the pilot study 
AT T s discriminant functions. Figure 12 depicts parallel results for the jus- 
tification section. It now seems abundantly clear that the two curves remain 
close together over the first five steps or so and theu diverge. The curve 
for the sample classified by its own discriminant functions continues to rise 
to some maximum point of classification accuracy, usually between Steps 40 and 
50. The best classification accuracy for the sample being classified by the 
other sample's discriminant functions is achieved very early, typically before 
Step 5 after' which the classification accuracy begins to deteriorate and the 
two curves diverge. Comparing Figure 11 with Figure 12 it can be seen again 
that better classification accuracy was achieved for both samples on the jus- 
tification section. Table 27 presents the classification matrices illustrat- 
ing this outcome more clearly. Of the 144 pilot study AT T s, 137 (95%) were 
classified correctly at Step 58 in the discriminant analysis of the justifica- 
tion section for the pilot study AT's. Of the 138 cross validation AT's, 91 
(66%) were classified correctly on the justification section with the pilot 
study AT T s discriminant functions, this maximum classification accuracy occur- 
ring on the f.iird step. 

Two cross validation studies also were conducted for the cross validation 
BT T s and the pilot study BT's, and the results from these studies continue to 
support the picture that has already emerged. Figure 13 portrays the accuracy 
of classifying the cross validation BT T s and the pilot study BT T s into correct 
criterion group on the evaluation section using the cross validation BT's dis- 
criminant functions. Parallel results for the justification section are shown 
in Figure 14. Perfect classification accuracy was achieved on the justifica- 
tion section at Step 46 for the 84 cross validation BT's using their own dis- 
criminant functions (see Table 28). Of the 80 pilot study BT T s, 54 (68%) were 
classified correctly on the justification section with the cross validation 
BT T s discriminant functions, this maximum classification accuracy occurring at 
Step 8. 

In the reverse cross validation for the pilot study BT T s and the cross 
validation BT T s classified by the pilot study BT's discriminant functions, the 
results are similar. Figure 15 portrays the curve of classification accuracy 
for the evaluation section and Figure 16 depicts the classification accuracy 
curve for the justification section. Table 29 presents the four classifica- 
tion matrices corresponding to Figures 15 and 16 in which the best classifica- 
tion achieved is portrayed. All 80 pilot study BT's were classified correctly 
on the justification section by their own discriminant functions. Of the 84 
cross validation BT's, 55 (65%) were classified correctly on the justification 
section with the pilot study BT's discriminant functions, this maximum clas- 
sification accuracy occurring at the first step. 

The results of the two cross validation studies were very similar for the 
total cross validation sample and the total pilot study sample, for the cross 
validation AT's and the pilot study AT's, and for the cross validation BT's 
and the pilot study BT's. These findings support the expectation held at the 
outset of this investigation that it would be possible to index the cross 
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TABLE 27 

BEST CLASSIFICATION INTO THE THREE CRITERION GROUPS 
USING THE PILOT STUDY AT's DISCRIMINANT FUNCTIONS 



EVALUATION SECTION - 19R 



PILOT STUDY AT's. (N=144) 



CROSS VALIDATION AT's (N=138) 



Step 50 



pi pu 

O *H 

m w UPPER 

CD M 
4J 0) 

H H MIDDLE 

a 

3 pu LOWER 

•M O 
O M 



Classification by 
Discriminant Analysis 

UPPER MIDDLE LOWER 



35 



28 



42 



Diagonal Sum = 105 



Step 3 



pi pu 

O -H 

Tl w UPPER 

4J 0) 

Tl S MIDDLE 

a 

rt pu LOWER 

u o 
V u 

< o 



Classification by 
Discriminant Analysis 



UPPER 
22 
22 
7 



MIDDLE 
7 
3_ 
2 



LOWER 
16 
19 
40 



Diagonal Sum = 65 



JUSTIFICATION SECTION - 19S 



PILOT STUDY AT's (N=144) 



CROSS VALIDATION AT's (N=138) 



Step 58 
pi Pu 

O iH 

ft 2 UPPER 
£ *§ MIDDLE 

^ §• LOWER 

i) o 
o u 



Classification by 
Discriminant Analysis 

UPPER MIDDLE LOWER 



45 
2 
0 



2 
36. 

0 



2 
1 

56 



Step 3 



Diagonal Sum = 137 
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LOWER 



Classification by 
Discriminant Analysis 



UPPER MIDDLE 
19 20 



8 
1 



30 
6 



LOWER 
6 
6 
42 



Diagonal Sum = 91 
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TABLE 28 

BEST CLASSIFICATION INTO THE THREE CRITERION GROUPS 
USING THE CROSS VALIDATION BT's DISCRIMINANT FUNCTIONS 



EVALUATION SECTION - 19R 



CROSS VALIDATION BT's (N=84) 



Step 62 



O tH 
•H ^ 

ft w UPPER 

4-» 0) 

£j 0 MIDDLE 

O CL) 

S 9* LOWER 

4-» O 
O M 

< O 



Classification by 
Discriminant Analysis 



UPPER 
29 
2 
0 



MIDDLE 
0 
28 
0 



LOWER 
0 
0 
25 



Step 8 



Diagonal Sum = £T2 



£ a, 

O tH 

M CO 

a) m 

4-> o 

•H 

o a) 

4-> O 

o u 



PILOT STUDY BT's (N=80) 

Classification by 
Discriminant Analysis 

UPPER MIDDLE LOWER 
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MIDDLE 

LOWER 



11 
12 
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14 
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JUSTIFICATION SECTION - 19S 
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UPPER MIDDLE LOWER 

19 3 4 

11 16 8 

0 0 19 

Diagonal Sum = 54 



5° 




in 




o 




LO 




00 




LO 




LO 




LO 




J- 




LO 




CM 




LO 




O 




LO 


CO 


00 


•H 




VS 


LO 




-d~ 


cd 




a 




<d 


CM 


4J 


-3* 


c 


O 


cd 




c 


00 


•H 


CO 


6 




•H 


LO 


V-i 


CO 


CJ 


ID 


cn 


CO 


•H 


J- 


P 


co 


0) 


CM 


CO 


CO 


•H 


o 


8. 


CO 




0) 


uO 


4J 


CM 




o 

CM 


<n 


UO 


th 


i-H 




J- 


c 


i-H 


•H 


CO 




i-H 




CM 




r-H 


■§ 


i-H 




i-H 




o 


a* 


i-H 


a) 




4J 


cn 




CO 









4J 

o 

CO tH 
CO «rl 
O 

a) jj pi 

■P bOrH 

c 

C CO 

id P cs 
o 

O 3 4J 

oo o a 

II M 03 

^ O cn 



CO 



c 
o 



a 
o 

- tH 
H H 4J 
PQ Q) id 

*H 

TJ V-i 
3 O 



4J 

cn 



o 



cd 

> 



•H O 

PL* CJ 

CD O 



4J 

a 

0) CO 



G 

•rl <f 

m II 

CO ^ 
CO 

id co 



MH 
O 



4J 
C 

cd 
a 

•H 

0 

•H 

Jh 
U 
CO 
•H 
P 

cn 



o - 

>■» *H H 
U 4J 

cd 

•H X) 
rH 3 

cd 
> 



cd 
u 

a 
u 
< 



PQ 



4-1 

cn 



0) 



/ 



CD 



GO 

>* 
TJ 

+- 

CO 



_o 

Q_ 



\ 



CM 

lo 
o 

LO 
CO 
LO 

LO 
LO 

LO 
CM 
LO 



00 
LO 

CM 

o 
oo 



LO 
CM 

O 
CM 



O 

r-H 

cn 
oo 

LO 
LO 





4J 






o 






CO rH 






CO «H 






O 






H 


cn 




o a) 








rH 












I 




4J 60 






C 


c 


CO 




o 


•H 


C CO 


•H 


CO 


cd p 


4J 






CJ 


rH 




0) 


cd 


O 3 


cn 


CJ 


00 O 




< 


It V-i 


C 




r3 O 


o 


4J 




•rl 


C 


c 


4J 


cd 


CO o 


cd 


c 


- *H 


CJ 


•rl 


H Jh 


•rl 


s 


PQ 0) 


MH 


•H 


4J 


•rl 


rH 


r>> «H 


4J 


u 


X) V-i 


CO 


CO 


3 o 




•H 






P 


cn -p 






u 




0) 


4J a) 


CO 


CO 


O V-i 


c 


•rl 


rH V-I 


o 


I 


•H O 


•H 




PM O 


4J 


0) 




CJ 


4J 


a) o 


C 


cn 


& 4J 






4J C 


PH 


0) 


•H 






60 


4J 


4J 




c 




•H <t 


cd 


»«* 


00 


c 


•H 


m ii 


•rl 




•H £3 


2 


M 


CO ^ 


•rl 




CO 


U 




cd co 


CJ 


■§ 


rH - 


CO 




O H 


•rl 


r3 


pq 


P 










o a 


CO 


0) 


o 




4J 


tH 


H 


cn 


CJ 4J 


PQ 




cd cd 






V-i 












CJ rH 






cj cd 


•U 




< > 


cn 



u 

60 
•H 



oo 



o o 

\D LO 



o o o 

CO CM r-H 



oo ooooo o 

00 LO LO CO CM i-H 



Number of Cases 
Classified Correctly 



Number of Cases 
Classified Correctly 



60 



TABLE 29 

BEST CLASSIFICATION INTO THF THREE CRITERION GROUPS 
USING THE PILOT STUDY BT T s DISCRIMINANT FUNCTIONS 



EVALUATION SECTION - 19R 



PILOT STUDY BT's (N=80) 



CROSS VALIDATION BT's (N=84) 



Step 50 



o 



♦H 

U B 

u o 

< o 



UPPER 



MIDDLE 



LOWER 



Classification by 
Discriminant Analysis 

UPPER MIDDLE LOWER 



24 
0 
1 



2 
0 



0 
0 
18 



Diagonal Sum = 77. 



Step 1 



•H 



0) 
4J 



U CD 

IS 

Ctf CX 

4-1 O 

CJ W 

< o 



UPPER 



MIDDLE 



LOWER 



Classification by 
Discriminant Analysis 

UPPER MIDDLE LOWER 



14 
10 
6 



3 
3 



12 
11 
16 



Diagonal Sum = 39 



JUSTIFICATION SECTION - 19S 



PILOT STUDY BT's (N=80) 



Step 40 



o 



0) 
4J 



U Q) 



UPPER 



MIDDLE 



§ §< LOWER 

4J O 
CJ U 

< o 



Classification by 
Discriminant Analysis 



UPPER MIDDLE 



26 
0 
0 



0 

0 



LOWER 
0 
0 
19 



Diagonal Sum = 80_ 



CROSS VALIDATION BT's (N=84) 



Step 1 



O tH 
•H 

0) M 

4J 0) 

u a 

U 0) 



UPPER 



MIDDLE 



§ ^ LOWER 

4J O 

u u 



Classification by 
Discriminant Analysis 

UPPER MIDDLE LOWER 



22 



14 



13 



1 4 20 

Diagonal Sum = 55_ 
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validation sample in the blind, without knowledge of criterion group member- 
ship, and achieve as good classification accuracy as was achieved with the pi- 
lot study sample where criterion group membership was known to the indexer. 
Further, it can be concluded that better classification into the three crite- 
rion groups using an optimum combination of the 67 quantitative variables is 
achieved when the two occupational ratings represented in the pilot study sam- 
ple and the crops validation sample are treated separately. The superior clas- 
sification accuracy achieved for the BT T s compared to the AT T s indicates that 
the Aviation Electronics Technician rating may represent a more varied amal- 
gamation of technical activities than the Boilerman rating. These findings 
suggest that classification procedures based on the content analysis methodolo- 
gy developed in this research should be tailored to specific occupations. In 
all of the results presented thus far, better classification was achieved in 
the discriminant analyses of th?. justification section compared to the evalua- 
tion section. In the various cross validation runs reported, it appears that 
the best classification accuracy that can be achieved on a second sample using 
discriminant functions developed on the first sample is 65 to 70 percent. 
This level of accuracy is achieved early in the stepwise discriminant analysis 
procedure, typically by the fifth step. This finding suggests that the varia- 
bles selected by the discriminant analysis program at the first five steps are 
crucial variables. Later in this section the variables selected for the first 
15 steps in each of the cross validation runs will be enumerated. But before 
turning to this enumeration, the results in classifying the generalization sam- 
ple (CS's and RM's) will be discussed. 

Discriminant analyses also were performed for the CS's and RM's combined 
(the total generalization sample) as well as for each of these two occupation- 
al ratings separately. The accuracy of classifying the generalization sample 
into correct criterion group is shown in Figures 17 and 18, first for the 
evaluation section and then for the justification section. Better classifica- 
tion accuracy was achieved for the justification section (see Table 30) than 
for the evaluation section where 190 of the 222 cases (86%) in the justifica- 
tion portion of the generalization sample were classified correctly. 

When the CS's and RM's were considered separately, better classification 
accuracy was achieved. Referring to Table 30, 58 of the 60 generalization 
CS's (97%) were classified correctly on the evaluation section. All 60 gener- 
alization CS's were classified correctly on the justification section. The 
curves depicting the accuracy of classifying the CS's on both the evaluation 
section and the justification section are shown in Figures 19 and 20, The 162 
RM's in the generalization sample were not classified as accurately as the CS's, 
although better classification accuracy for the RM's was achieved on the justi- 
fication section where 144 cases were classified correctly (89%), These re- 
sults are shown in Table 30, The curves portraying the accuracy of classifying 
the RM's on both the evaluation section and the justification section are shown 
in Figures 21 and 22. Again the superiority of the justification section in 
providing discriminating narrative comments is demonstrated. On the justifica- 
tion section all 60 CS's were classified correctly by their discriminant func- 
tions and 89 percent of the RM's were classified correctly by their discrimi- 
nant functions. Eighty-six percent of the total generalization sample was 
classified correctly oa the justification section by the discriminant functions 
based on CS's and RM's combined. This suggests that the occupational rating, 




Numb r of Cases Classified Correctly 




Number of Cases Classified Correctly 
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TABLE 30 

BEST CLASSIFICATION INTO THE THREE CRITERION GROUPS 
FOR THE GENERALIZATION SAMPLE 



EVALUATION SECTION 



19R 



JUSTIFICATION SECTION 



LOS 



CS T s & RM's COMBINED (N=222) 



Step 61 
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Classification by 
Discriminant Analysis 

UPPER MIDDLE LOWER 
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Diagonal Sum = 170 



CS's & RM T s COMBINED (N=222) 

Step 54 Classification by 

Discriminant Analysis 
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Radioman, may be more heterogeneous and require a greater variety of skills 
than the Commissaryman rating just as the Aviation Electronics Technician rat- 
ing may .represent a more varied amalgamation of technical activities than the 
Boilerman rating. The case for treating each occupational rating separately 
seems to be supported additionally by the classification results for the gen- 
eralization sample* Moreover, it is evident that the content analysis method- 
ology developed initially on the pilot study sample consisting of AT 1 . s and 
BT's was generalizable to a new sample consisting of two different occupation- 
al ratings, viz, , CS's and RM T s. 

A most interesting set of results is revealed by an examination of which 
variables were selected by the stepwise discriminant analysis program for the 
first 15 steps in each of the computer runs that were made. Tables 31 and 32 
show the results for the total cross validation sample (AT's and BT's combined) 
and the total pilot study sample (AT T s and BT T s combined). The results are 
presented separately for the evaluation section and the justification section. 
If the variable was one of the first 15 variables selected in both of the sam- 
ples for each section considered separately,, it has been flagged with an as- 
terisk in Tables 31 and 32. Therefore, it can be determined quickly that six 
of the first 15 variables selected by the stepwise discriminant analysis pro- 
gram in the cross validation sample and the pilot study sample for the evalua- 
tion section were the same, namely, Total Number of 5 (New 3) Weights, Total 
Number of 2 (New -1) Weights, f of COOPERATION, wf of AWARDS AND PUNISHMENT, 
f of LEADERSHIP AND DIRECTING, and wf of TECHNICAL SKILLS. Furthermore, Total 
Number of 5 (New 3) Weights was the first variable selected in. both samples 
for the evaluation section, with Total Number of 2 (New -1) Weights being the 
second variable selected in the pilot study sample and the third variable se- 
lected in the cross validation sample. It appears that the incidence of super- 
lative adjectives and adverbs describing excellent performance (5 Weights) and 
the incidence of negatively comparative adjectives and adverbs describing in- 
adequate performance (2 Weights) constitute key discriminating variables in 
analyzing the content of narrative comments written for the evaluation section 
in these two samples. 

Of more interest is the justification section since without exception 
better classification was achieved in the content analysis of the narrative 
comments written for the justification section. Four of the first 15 varia- 
bles selected by the stepwise discriminant analysis program for the justifica- 
tion section in the total. cross validation sample (AT's and BT's combined) and 
the total pilot study sample (AT's and BT's combined) were the same. Total 
Number of Index Terms Used was the first variable selected in both samples. 
Sum of Variables 1 through 29 was the second variable selected in the cross 
validation sample and the fourth variable selected in the pilot study sample, 
wf of PRODUCTIVITY AND ACHIEVEMENT was the third variable selected in the pi- 
lot study sample and the fourth variable selected in the cross validation sam- 
ple, f of DRIVE was the fifth variable selected in the pilot study sample and 
the seventh variable selected in the cross validation sample. 

Similar tables of results also were prepared for the AT's and BT T s con- 
sidered separately in the cross validation and pilot study samples. Tables 33 
and 34 show the first 15 variables selected by the stepwise discriminant analy- 
sis program for the cross validation AT's and the pilot study AT's in the 
evaluation and justification sections. As in Tables 31 and 32, an asterisk 
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TABLE 31 

VARIABLES SELECTED BY THE STEPWISE DISCRIMINANT ANALYSIS PROGRAM 
AT STEPS 1 THROUGH 15 FOR THE CROSS VALIDATION SAMPLE 



Step Evaluation Section 

1 Total Number of 5 (New 3) 

Weights* 

2 Total Number of Index Terms 

Used 

3 Total Number of 2 (New -1) 

Weights* 

4 f of COOPERATION* 



5 f of PRODUCTIVITY AND ACHIEVE- 
MENT 

.. 6 wf of RESOURCEFULNESS 

7 f of PROFESSIONALISM 

8 wf of AWARD'S AND PUNISHMENT* 

9 Total Number of 4 (New 2) 

Weights 

10 f of LEADERSHIP AND DIRECTING* 

11 wf of RESPONSIVENESS 

12 f of TECHNICAL. SKILLS 

13 f of POTENTIAL 

14 >Tf of TECHNICAL SKILLS* 

15 f of RESPONSIVENESS 



Justification Section 

Total Number of Index Terms 
Used* 

Sum of Variables 1 through 29* 



wf of ORGANIZATION 



wf of PRODUCTIVITY AND ACHIEVE- 
MENT* 

f of ENDURANCE 1 



'wf of TECHNICAL SKILLS 
f of DRIVE* 

f of INTELLECTUAL FUNCTIONING 
f of ASSET TO THE NAVY 

wf of POTENTIAL 

f of TECHNICAL SKILLS 

f of CONDUCT, INTEGRITY, AND 
PRIDE 

wf of AWARDS AND PUNISHMENT 
f of LEADERSHIP AND DIRECTING 
wf of LEADERSHIP AND DIRECTING 



Also selected by the stepwise discriminant analysis program for this section 
in the pilot study sample. 
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TABLE 32 

VARIABLES SELECTED BY THE STEPWISE DISCRIMINANT ANALYSIS PROGRAM 
AT STEPS 1 THROUGH 15 FOR THE PILOT STUDY SAMPLE 



Step Evaluation Section 

1 Total Number- of 5 (New 3) 

Weights* 

2 Total Number of 2 (New -1) 

Weights* 

3 f of LEADERSHIP AND DIRECTING* 



A wf of TECHNICAL SKILLS* 

5 Total Number of 1 (New -2) 

Weights 

6 wf of AWARDS AND PUNISHMENT* 



7 wf of MANAGEMENT FUNCTIONS 

8 f of SERVICE MOTIVATION 

9 f of COOPERATION* 

10 Total Number of 3 (New 1) 

Weights 

11 wf of REPRESENTATION 

12 wf of CONTROLLISc 

13 wf of COMMUNICATION 

14 wf of INITIATIVE 

15 f of INITIATIVE 



Justification Section 

Total Number of Index Terms 
Used* 

Sum of Variables 31 through 59 

wf of PRODUCTIVITY AND ACHIEVE- 
MENT* 

Sum of Variables 1 through 29* 
f of DRIVE* 

Total Number of 3 (New 1) 
Weights 

wf of STAFFING 

f of RESOURCEFULNESS 

wf of SERVICE MOTIVATION 

f :f REPUTE 

wf of ASSET TO THE NAVY 
wf of FLEXIBILITY 
wf of GROOMING AND ATTIRE 
f of AWARDS PUNISHMENT 
wf of .CONTROLLING 



Also selected by the stepwise discriminant: analysis program for this section 
in the cross validation sample. 
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TABLE 33 - 

VARIABLES SELECTED BY THE STEPWISE DISCRIMINANT ANALYSIS PROGRAM 
AT STEPS 1 THROUGH 15 FOR THE CROSS VALIDATION AT 1 s 



Step Evaluation Section 

1 f of COOPERATION* 



Total Number of 5 (New 3) 
Weights* 

f of TECHNICAL SKILLS 



4 f of LEADERSHIP AND DIRECTING 

5 Tonal Number of 2 (New -1) 

Weights* 

6 wf of RESP0NS7 7ENESS 

7 f of RESOURCEFULNESS 

8 f of PRODUCTIVITY AND ACHIEVE- 

MENT* 

9 f of SERVICE MOTIVATION 

10 wf of TECHNICAL SKILLS 

11 Total Number of 4 (New 2) 

Weights 

12 f of AWARDS AND PUNISHMENT 

13 f of PROFESSIONALISM 

14 Total Number of Index Terms 

Used* 

15 Total Number of Words in Text 



Justification Section 

Total Number " Index Terms 
Used* 

wf of TECHNICAL SKILLS 



wf of PRODUCTIVITY AND ACHIEVE- 
MENT 

wf of SKILLS AND ABILITIES 
f of COMMUNICATION 

wf of ORGANIZATION 

wf of ASSET TO THE NAVY 

wf of REPUTE 

wf of AWARDS AND PUNISHMENT 
f of INTELLECTUAL FUNCTIONING* 
wf of POTENTIAL 

f of REPRESENTATION 

Total Number of Words in Te:^ 

wf of Au^POSSIVENESS 

f of RESPONSIVENESS 



Also selected by the stepwise discriminant analysis program for this section 
ir Hie pilot study AT subsample. 
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TABLE 34 

VARIABLES SELECTED BY THE STEPWISE DISCRIMINANT ANALYSIS PROGRAM 
AT STEPS 1 THROUGH 15 FOR THE PILOT STUDY AT's 



Step Evaluation Section 

1 Total Number of 5 (New 3) 

Weights* 

2 Total Number of 2 (New -1) 

Weights* 

3 wf ol: AWARDS AND PUNISHMENT 



4 f of COMMUNICATION 

5 f of PRODUCTIVITY AND ACHIEVE- 

MENT* 

6 f of COOPERATION* 

7 wf of MANAGEMENT FUNCTIONS 

8 wf of PRODUCTIVITY AND ACHIEVE- 

MENT 

9 wf of SERVICE MOTIVATION 

10 Tctal Number of Index Terms 

Used* 

11 of USE OF COMMUNICATION 

12 f of USE OF COMMUNICATION 

13 wf of REPRESENTATION 

14 wf of GROOMING AND ATTIRE 

15 wf of DRIVE 



Justification Section 

Total Number of Index Terms 
Used* 

Total Number of 5 (New 3) 
Weights* 

f of CONDUCT, INTEGRITY, AND 
PRIDE 

f of INTELLECTUAL FUNCTIONING* 
wf of INTELLECTUAL FUNCTIONING 

wf of GROOMING AND ATTIRE 
f of TECHNICAL SKILLS 
f of REPUTE 

wf of FLEXIBILITY 

f of PRODUCTIVITY AND ACHIEVE- 
MENT 

Sum of Variables 1 through 29 

wf of ENDURANCE 

wf of SERVICE MOTIVATION 

wf of CONTROLLING 

f of ENDURANCE 



Also selected by the stepwise discriminant analysis program. for this, section 
in the cross validation AT subsample. 
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denotes variables within each section that were selected in both samples. On 
the evaluation section, five of the first 15 variables selected were the same 
for the cross validation AT's and the pilot study AT f s. As expected, Total 
Number of 5 (New 3) Weights and Total Number of 2 (New -1) Weights were among 
these five variables. On the justification section, two of the first 15 varia- 
bles selected were the same for the cross validation AT f s and the pilot study 
AT ! s. Again, Total Number of Index Terms Used was the first variable selected 
for both the cross validation AT f s and the pilot study AT f s. 

Tables 35 and 36 show the first 15 variables selected by the stepwise dis- 
criminant analysis program for the cross validation BT T s and the pilot. study 
BT's in the evaluation and justification sections. Again, an asterisk denotes 
variables within each section that were selected in both samples. On the 
evaluation section, two of the first 15 variables selected were the same for 
the cross validation BT's and the pilot study BT's. Total Number of 5 (New 3) 
Weights was the first variable selected for both the cross validation BT's and 
the pilot study BT's. On the justification section, five of ; the first 15 varia 
bles selected were the same for the cross validation BT f s and the pilot study 
BT f s. As with the two AT subsamples, Total Number of Index Terms Used was the 
first variable selected for both the cross validation BT's and the pilot study 
BT's. 

The key discriminating variables for the cross validation and pilot study 
samples as well as for the AT and BT subsamples were Total Narber of 5 (New 3) 
Weights and Total Number of 2 (New -1) Weights in the evaluation section. In 
the justification section without exception the key discriminating variable 
was Total Number of Index Terms Used. What are the results like for the gener- 
alization sample? Table 37 shows that these same three variables were those 
selected first in the stepwise discriminant analysis of the generalization sam- 
ple. Total Number of 2 (New -1) Weights and Total Number of 5 (New 3) Weights 
were selected first and second in the evaluation section. Total Number of In- 
dex Terms Used was selected first in the justification section. 

When the two occupational groups comprising the generalization sample are 
considered individually, the results are not quite unanimous. Neither Total 
Number of 5 (New 3) Weights nor Total Number of 2 (New -1) Weights was select- 
ed in the evaluation section for the generalization CS 1 s (se' Table 38). How- 
ever, Total Number of Index Terms Used was the first variable selected in the 
justificati*. , section for the generalization CS's. In Table 39 showing the 
resjlts for the generalization RM's, Total Number of 2 (New -1) Weights was 
selected as the first variable in the evaluation section and Total Number of 5 
(New 3) Weights was selected as the fifth variable. Again, Total Number of 
Index Terms Used was selected first in the justification section. 

Two findings are striking. Without exception better classification was 
achieved in the content analysis of the narrative comments in the justifica- 
tion section, and without exception the first variable selected in the various 
stepwise discriminant . ialyses for the justification section was Total Number 
of Index Terms Used. This variable reflects the variety of specific areas of 
a ratee's performance that the evaluator chose to comment on, and is measured 
by the' number of different index terms selected by the indexer to encompass 
the narrative content. It appears that the range of skills and abilities that 
a chief petty officer possesses may be a key factor in his superior performance 
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TABLE 35 

VARIABLES SELECTED BY THE STEPWISE DISCRIMINANT ANALYSIS PROGRAM 
AT STEPS 1 THROUGH 15 FOR THE CROSS VALIDATION BT r s 



Step Evaluation Section 

1 Total Number of 5 (New 3) 

Weights* 

2 Total Number of Index Terms 

Used 

3 Sum of Variables 31 through 59 

4 wf of RESOURCEFULNESS 

5 f of COOPERATION 

6 f of RESOURCEFULNESS 

7 Total Number of Words in Text 

8 f of AWARDS AND PUNIf HMENT 

9 f of ORGANIZATION 

10 f of TECHNICAL SKILLS 

11 f of ASSET TO THE NAVY 

12 wf of ASSET TO THE NAVY* 

13 wf of FLEXIBILITY 

14 wf of REPUTE 

15 f of STAFFING 



Justii cation Section 

Total Number of Index Terras 
Used* 

wf of LEADERSHIP AND DIRECTING 



Sum of Variables 1 through 29 

f of PRODUCTIVITY AND ACHIEVE- 
MENT* 

wf of CONDUCT, INTEGRITY, AND 
PRIDE 

f of DRIVE 

wf of SERVICE MOTIVATION 

f of INITIATIVE* 

f of RESPONSIVENESS 

f of ASSET TO THE NAVY* 

wf of DRIVE 

wf of RESOURCEFULNESS 

f of RESOURCEFULNESS* 

f of PROFESSIONALISM 

f of REPRESENTATION 



Also selected by the stepwise discriminant analysis program for this section 
in the pilot study BT subsample. 
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TzvBLE : 

VARIABLES SELECTED BY THE STEPWISE 
AT STEPS 1 THROUGH 15 FOR 

Step Evaluation Section 

1 Total Number of 5 (New 3) 

Weights 

2 Total Number of 2 (New -1) 

Weights 

3 f of LEADERSHIP AND DIRECTING 

4 wf of PROFESSIONALISM 

5 f of responsiveness' 

6 wf of GROOMING AND ATTIRE 

7 wf of TECHNICAL SKILLS 

8 wf of CONTROLLING 

9 f of SKILLS AND ABILITIES 

10 wf of RELIABILITY AND DEPENDA- 

BILITY 

11 wf of MANAGEMENT FUNCTIONS 

12 wf of USE OF COMMUNICATION 

13 wf of ASSET TO THE NAVY* 

14 f of MANAGEMENT FUNCTIONS 

15 ; : of STAFFING 

* 

Also selected by the stepwise discrim 
in the cross validation BT subsample. 



6 

DISCRIMINANT ANALYSIS PROGRAM 
THE PILOT 3TUDY BT T s 

■te stification Section 

Tot. Number of Index Terras 
Used* 

wf of PRODUCTIVITY AND ACHIEVE- 
MENT 

f of AWARDS AND PU 'ISHMENT 

Sum of Variables 31 through 59 

f of REPUTE 

f of RESOURCEFULNESS* 

f of PRODUCTIVITY AND ACHIEVE- 
MENT* 

wf of CONTROLLING 

f of SERVICE MOTIVATION 

f of ASSET TO THE NAVY* 

f of ENDURANCE 

Total Number of 3 (New 1) 
Weights 

wf of uEPUTE 

f of USE OF COMMUNICATION 
fof INITIATIVE* 

ant analysis program for this section 
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TABLE 37 

VARIABLES SELECTED BY THE STEPWISE DISCRIMINANT ANALYSIS PROGRAM 
AT STEPS 1 THROUGH 15 FOR THE GENERALIZATION SAMPLE 



Step Evaluation Section 

1 Total Number of 2 (New -1) 

Weights 

2 Total Number of 5 (New 3) 

Weights 

3 wf of POTENTIAL 

k f of COMMUNICATION 

5 wf of MANAGEMENT FUNCTIONS 

6 f of RELIABILITY AND DEPENDA- 

BILITY 

7 f of INTELLECTUAL FUNCTIONING 

8 wf of SERVICE MOTIVATION 

9 f of ORGANIZATION 

10 f of RESPONSIVENESS 

11 wf of INITIATIVE 

12 wf of PLANNING 

13 wf of RELIABILITY AND DEPENDA- 

BILITY 

14 f of REPRESENTATION 

15 wf of STAFFING 



Justification SectLon 

Total Number of Index Terms 
Used 

f of COOPERATION 



wf of STAFFING 
wf of COMMUNICATION 
wf of PLANNING 
wf of POTENTIAL 

f of PRODUCTIVITY AND ACHIEVE- 
MENT 

wf of PRODUCTIVITY AND ACHIEVE 
MF.NT 

Sum of Variables 31 through 59 
wf of GROOMING AND ATTIRE 
f of ENDURANCE 
f of REPUTE 

wf of INTELLECTUAL FUNCTIONING 

f of RESPONSIVENESS 
wf of REPUTE 



.A 
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TABLE 38 

VARIABLES SELECTED BY THE STEPWISE DISCRIMINANT ANALYSIS PROGRAM 
AT STEPS 1 THROUGH 15 FOR THE GENERALIZATION CS ? s 

Step Evaluation Section Justification Section 

1 wf of CONTROLLING Total Number of Index Terms 

Used 

2 wf of ASSET TO THE NAVY ' f of PROFESSIONALISM 

3 f of MANAGEMENT FUNCTIONS f of INITIATIVE 

4 wf of SKILLS AND ABILITIES wf of COMMUNICATION 

5 Total Number of Index Terms wf of REPRESENTATION 

Used 

6 f of ENDURANCE wf of COOPERATION < 

7 Sum of Variables 31 through 59 f of CONTROLLING 

8 wf of ORGANIZATION f of POTENTIAL 

9 wf of POTENTIAL f of PRODUCTIVITY AND ACHIEVE- 

MENT 

10 wf of PLANNING wf of SKILLS AND ABILITIES 

11 f of SERVICE MOTIVATION wf of PLANNING 

12 f of PLANNING Total Number of Words in Text 

13 Total Number of 3 (New 1) wf of PROFESSIONALISM 

Weights 

14 f of LEADERSHIP AND DIRECTING f of SKILLS AND ABILITIES 

15 f of INITIATIVE f of RELIABILITY AND DEPENDA- 

BILITY 
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TABLE 39 

VARIABLES SELECTED BY THE STEPWISE DISCRIMINANT ANALYSIS PROGRAM 
AT STEPS 1 THROUGH 15 FOR THE GENERALIZATION RM f s 



Step Evaluation Section 

1 Total Number of 2 (New -1) 

Weights 

2 wf of AWARDS AND PUNISHMENT 

3 wf of POTENTIAL 

4 wf of MANAGEMENT FUNCTIONS 

5 Total Number of 5 (New 3) 

Weights 

6 f of RELIABILITY AND DEPENDA- 

BILITY 

7 f of COMMUNICATION 

8 f of POTENTIAL 

9 wf of REPUTE 

10 f of REPUTE 

11 f of INTELLECTUAL FUNCTIONING 

12 f of RESPONSIVENESS 

13 wf of RESPONSIVENESS 

14 wf of SERVICE MOTIVATION 

15 wf of PRODUCTIVITY AND ACHIEVE 

MENT 



Justification Section 

Total Number of Index Terms 
Used 

Sum of Variables 31 through 59 

wf of PRODUCTIVITY AND ACHIEVE- 
MENT 

f of PRODUCTIVITY AND ACHIEVE- 
MENT 

f of COOPERATION 

wf of GROOMING AND ATTIRE 

wf of STAFFING 

wf of ENDURANCE 

f of STAFFING 

wf of PLANNING 

Total Number of 4 (New 2) 
Weights 

f of POTENTIAL 

f of INITIATIVE 

wf of TECHNICAL SKILLS 

f of REPUTE 
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Another important finding is that better classification was achieved when 
each of the four occupational ratings studied in this research project was con- 
sidered individually* In addition to Total Number of 5 (New 3) Weights, Total 
Number of 2 (New -1) Weights, and Total Number of Index Terms Used, a number - 
of specific index terms among the first five selected should be called out as 
potential key variables in discriminating between superior chief petty offi- 
cers and their slightly less qualified colleagues. For the cross validation 
AT's these variables were COMMUNICATION; COOPERATION; LEADERSHIP AND DIRECTING; 
PRODUCTIVITY AND ACHIEVEMENT; TECHNICAL SKILLS; and SKILLS AND ABILITIES. For 
the pilot study AT f s these variables were AWARDS AND PUNISHMENT; COMMUNICATION; 
CONDUCT, INTEGRITY, AND PRIDE; INTELLECTUAL FUNCTIONING; and PRODUCTIVITY AND 
ACHIEVEMENT. The potential key variables for the cross validation BT's were 
CONDUCT, INTEGRITY, AND PRIDE; COOPERATION; LEADERSHIP AND DIRECTING; PRODUC- 
TIVITY AND ACHIEVEMENT; and RESOURCEFULNESS. For the pilot study BT's these 
variables were AWARDS AND PUNISHMENT; LEADERSHIP AND DIRECTING; PRODUCTIVITY 
AND ACHIEVEMENT; PROFESSIONALISM; REPUTE; and RESPONSIVENESS. The potential 
key variables for the CS's were ASSET TO THE NAVY; COMMUNICATION; CONTROLLING; 
INITIATIVE; MANAGEMENT FUNCTIONS; PROFESSIONALISM; REPRESENTATION; and SKILLS 
AND ABILITIES. For the RM T £ these variables were AWARDS AND PUNISHMENT; COOP- 
ERATION; MANAGEMENT FUNCTIONS; POTENTIAL; and PRODUCTIVITY AND ACHIEVEMENT. 
These same specific index terms were also the variables achieving statistical 
significance beyond the .001 level of probability in the Mann-Whitney U test 
as well as the t test of mean difference for the comparison on the cross vali- 
dation and generalization samples between the Middle and Upper criterion groups, 
the mo&t difficult discrimination to be made between any two of the three cri- 
terion groups (see Tables 20 to 23). 

Ten of the 29 index terms do not appear to lend as much assistance in the 
discrimination task as the key variables cited above. These less useful terms 
are the following: DRIVE; ENDURANCE; FLEXIBILITY; GROOMING AND ATTIRE; ORGANI- 
ZATION; PLANNING; RELIABILITY AND DEPENDABILITY; SERVICE MOTIVATION; STAFFING; 
and USE OF COMMUNICATION. However, all but two of these ten terms were select- 
ed as one of the sixth to the tenth variables in at least one of the stepwise 
discriminant analyses performed. The two exceptions were FLEXIBILITY and USE 
OF COMMUNICATION, the 71 east useful terms used in this study. These findings 
suggest that a smaller number of dimensions than the full complement of 67 
quantitative variables derived from the indexing procedure can be used to iden- 
tify superlative CPO's whose superior performance recommends them as candidates 
for promotion to a higher level of responsibility. 




SECTION 5. RELIABILITY STUDY 

A comprehensive reliability study was conducted whose objectives were two- 
fold: (1) to determine the level cf agreement among several individuals all 
of whom independently would perform a content analysis of the same corpus of 
Evaluation Reports, and (2) to investigate if nonresearchers could be trained 
successfully to apply the complex content analysis methodology developed in 
the pilot study. 

A set of 48 Evaluation Reports was selected by the Navy Personnel Research 
and Development Center, representing a cross section of the kinds of reports 
included in the overall experimental design for the cross validation and gen- 
eralization samples. In each of these 48 Evaluation Reports the evaluation 
section was separated from the justification section so that the narrative com- 
ments for each section were not considered together. This resulted in a group 
of 96 randomized pieces of narrative text to be indexed in the reliability 
study. To each of these 96 pieces of narrative text was appended the corre- 
sponding sections 4A and 4B of the Evaluation Report form. These two sections 
provide a description of the ratee's primary and collateral duties and should 
be read as background information before beginning to index the narrative text. 
Each of these 96 minidocuments was assigned a 4-digit identification number by 
NPRDC, Xeroxed in multiple copies, and sent to R-K Research and System Design 
to be used as the data base in the reliability study. 

Four individuals participated in the reliability study: (1) the experi- 
enced indexer who also indexed the pilot study sample, the cross validation 
sample, and the generalization sample; (2) the principal investigator; (3) 
an inexperienced indexer (inexperienced indexer A) with two years of college 
in the liberal arts; and (4) another inexperienced indexer (inexperienced in- 
dexer B) with executive secretary experience. To this end a training*; manual 
was prepared by the experienced indexer and the principal investigator to 
assist the two neophyte indexers in understanding their assignment. The train- 
ing manual in its entirety is included in this report as Appendix A. The ver- 
sion of the training manual shown in Appendix A is not the original version 
that was used to train the four reliability indexers, but rather is an updated 
version that includes voluminous examples of how to handle difficult indexing 
decisions and which also attempt to eliminate areas of confusion brought to 
light in analyzing the results of the reliability study. 

Six intensive training sessions were conducted by the experienced indexer 
in order to try to bring all four indexers up to a common level of expertise 
before beginning the actual study. Obviously, this objective could only be 
met partially in view of the varying educational backgrounds of the four relia- 
bility indexers and their different levels of previous exposure to the indexing 
dictionary. The study itself proved to be a traumatic experience for the three 
reliability indexers who had not spent the past year doing the actual indexing 
of the cross validation and generalization samples, probably because they were 
still in the early stages of their learning curves. The task given to each 
participant was extremely difficult and can be likened to a take-home, open- 
book final examination without a time limit. However, given the experience of 
participating in the study, all four of the reliability indexers now feel that 
they are better qualified to function as regular indexers and could perform . 
this assignment in a consistent manner. 
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When all four reliability indexers had completed indexing the 96 nieces 
of narrative text, their indexing decisions were recorded side by side on 
work sheets for each segment of narrative text indexed. These work sheets 
provided the data base for computing agreement statistics. In all of the sta- 
tistical computations reported subsequently in this section, assignment of 
the index terms was considered to be a separate intellectual task from assign- 
ing the corresponding weights based on the modifying adjectives and adverbs, 
^here is good justification for analyzing the reliability study results in 
these two contexts. When an indexer studied a segment of narrative text, the 
first step was to select an appropriate index term or terms from among the 29 
possibilities that best described the substantive content of the text* Once 
the indexer had completed this first phase of the concent analysis^ then the 
segment of narrative text was rescanned to identify the adjectives and adverbs 
that defined the numerical weight to be assigned to e 1 index term chosen* 
Considering these judgments as two sequential decision processes also made 
the results of the reliability study in :e amenable to statistical analysis as 
will be shown in the subsequent discussion. 

As early as 1960 Cohen, in introducing a new agreement statistic called 
kappa, pointed out that for most problems in nominal scale agreement between 
two judges or decision makers, many investigators compute a contingency chi 
square as a test of the hypothesis of chance agreement, and some investigators 
have gone on to compute the contingency coefficient, C, as a measure of degree 
of agreement.^ However, Cohen concluded that the use of chi square (x 2 ) » and 
therefore, the C which is based on it for the evaluation of agreement is inde- 
fensible. When applied to a contingency table, x^ tests the null hypothesis 
with regara to association, not agreement. Therefore, x^ and C are inappro- 
priate statistics for measuring agreement since they will be inflated quite 
impartially by any departure from chance association, either disagreement or 
agreement. In order to remedy this situation, Cohen suggested a new coeffi- 
cient, kappa, to measure the degree of agreement in nominal scales, and to 
provide means for testing hypotheses and setting confidence limits for this 
coefficient . 

Quoting from Cohen's 1960 article [20, pp. 39-40], "...for any problem in 
nominal scale agreement between two judges, there are only two relevant quan- 
tities: 

p^ = the proportion of units in which the judges agreed 

= the proportion of units for which agreement is expected by chance. 

The test of agreement comes then with regard to the 1 - p of the units 
for which the hyprchesis of no association would predict disagreement between 
the judges. This term will serve as the denominator. 

"To the extent to which nonchance factors are operating in the direction 
of agreement, p will exceed p ; their difference, p - p , represents the pro- 
portion of the cases in which fSeyond-chance agreement occurred and is the nu- 
merator of the coefficient. 

"The coefficient < is simply the proportion of chance-expected disagree- 
ments which do not occur, or alternatively, it is the proportion of agreement 
after chance agreement is removed from consideration: 
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The significance of an obtained k is determined by dividing k by a 



where a = v / —7- *- , The resulting critical ratio is referred to the 

V N(l - p ) 0 

normal curve. However, Cohen has pointed out that it is generally of as little 
value to test k for significance as it is for any other reliabilitv coefficient 

to know merely that k is beyond chance is trivial since one usually expects 

much more than this in the way of reliability in psychological measurement. 
However, the size of the critical ratio does provide some immediate feedback 
concerning the magnitude of the agreement achieved beyond the level expected 
by chance. Probably a. more useful way to interpret the significance of an 
obtained k is in terms of the maximum value of k. The theoretical upper limit 
of k is +1,00, but this limit can only be reached if the off-diagonal (dis- 
agreement) cells in the agreement matrix are all zero. This in turn demands 
that the marginal probabilities for each diagonal (agreement) cell must be 
identical. Perfect agreement between two judges is rarely achieved, and there- 
fore, the marginal distributions in any agreement matrix are not identical. 
This means that in practice the upper limit of k is never +1.00 but rather some 
lesser value. The maximum value of k is set by the marginal distributions in 
any particular application of the kappa agreement statistic, and :».t can be cal- 
culated. A comparison of the obtained k with its maximum upper limit computed 
from the marginal distributions provides the investigator with a more useful 
index of how closely f'^e agreement level that was achieved between two judges 
approached the maximuu. level of agreement that was possible. 

The kappa statistic was the measure of agreement used in analyzing the in- 
dex terms assigned by the four reliability indexers. For each segment of nar- 
rative text, each indexer chose a term or terms from the list of 29 possibili- 
ties, or the decision was made that no term should be used. From a careful 
analysis of these indexing decisions for each pair of reliability indexers, six 
pairwise agreement matrices were constructed. These were 30 by 30 matrices, 
with the 29 index terms representing 29 of the 30 nominal categories and No In- 
dex Term Used representing the 30th nominal category. The pairwise indexing 
decisions for each segment of narrative text analyzed across all 96 documents 
in the reliability study data base were tabulated into the appropriate cell of 
the agreement matrix for the particular pair of indexers being compared. The 
30 diagonal cells of the agreement matrix denote agreement between the two in- 
dexers in assigning index terms; all of the off-diagonal elements in the matrix 
represent instances in which the two indexers disagreed in their selection of 
terms. The total number of entries in these six matrices varied slightly among 
the six pairvi^'tibmparisons between the four reliability indexers, but in all 
instances they, were^v&ry large, ranging from 1,230 tallies to 1,389 tallies. 
Consequently, the^ si^e of| the reliability study data base can be considered to 
be large enough to provide a stable measure of the level of agreement achieved 
in performing this complex intellectual task. 

Table 40 shows the results of the kappa analysis of the six pairwise com- 
parisons between the four reliability indexers in selecting index terms for 
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TABLE 40 

RESULTS OF THE KAPPA ANALYSIS FOR THE SIX PAIRWISE COMPARISONS 
BETWEEN THE FOUR RELIABILITY INDEXERS IN SELECTING INDEX TERMS 
FOR THE ENTIRE RELIABILITY STUDY DATA BASE 



Pairwise Comparisons 
Between Each Pair of 
Reliability Indexers 

The experienced indexer vs, 
the principal investigator 



K 

,73 



,0065 



111.82 



max 



.90 



The experienced indexer vs. 
inexperienced indexer A 



.88 



,0072 



123.49 



.97 



The experienced indexer vs. 
inexperienced indexer B 



,72 



,0068 



106.24 



.90 



The principal investigator 
vs. inexperienced indexer A 



.73 



.0065 



111.82 



.89 



The principal investigator 
vs. inexperienced indexer B 



,71 



.0064 



111.06 



.90 



Inexperienced indexer A vs. 
inexperienced indexer B 



.78 



,0068 



115.56 



.92 



A z of 3.29 is significant at the .001 level of probability. Therefore, all 
of the z values reported in this table are extremely significant and lead to 
rejection of the null hypothesis that the obtained k uoes not exceed the 
chance level of agreement. 



the entire reliability study data base. The second column in this table sh-jws 
the value of k; the third column shows the standard error of k; and the fourth 
column lists the normal deviate, z, obtained by dividing k by its standard er- 
ror. All of the z values are very large, and consequently, extremely signifi- 
cant, indicating that in all six comparisons the null hypothesis that the ob- 
tained k does not exceed the chance level of agreement can be rejected. The 
last column in Table 40 provides the maximum possible value of kappa for each 
of he six pairwise comparisons. Inese values can be used as an upper limit 
for omparing the level of agreement actually achieved with the maximum level 
post ble given the marginal distributions. Thus, in the first comparison be- 
twee the experienced indexer and the principal investigator, the k obtained 
was -73 compared to a possible maximum value of .90. The best agreement in 
selecting index terms was obtained between the experienced indexer and inex- 
perienced indexer A, a k of .88 where the maximum k possible in this instance 
was .97. Thin is a heartening finding, suggesting that an individual without 
a research background in only six training sessions can be trained to apply 

O 
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♦"he complex content analysis methodology developed in this research project, 
with additional training and further experience, this individual could be ex- 
pected to apply these indexing skills in an even more consistent manner. 

The other values of K in Table 40 are not as large as the one for the com- 
parison between the experienced indexer and inexperienced indcxer A. However, 
they all range between .71 and .78, with .71 probably representing the lower 
limit of reliability achievable in a study of this kind. With additional 
training in those areas where there was confusion in the minds of the relia- 
bility indexers as they wrestled .*it:h the task of selecting the most appropri- 
ate index term, the expectation would be that better agreement could be 
achieved among these same four individuals if they were to replicate this ex- 
periment. All four reliability indexers have expressed their concordance with 
this expectation. 

A careful perusal of the six agreement matrices from which the statistics 
presented in Table 40 were derived was very instructive. Most of the off- 
diagonal cells were empty. When there were tallies, they were sporadic and 
scattered with only one or two tallies appearing in an occasional cell off the 
diagonal. However, two areas of confusion were prominently displayed in these 
six matrices. One minor area of confusion was between PRODUCTIVITY AND ACHIEVE- 
MENT and MANAGEMENT FUNCTIONS, although this confusion could not be considered 
to be extensive. The revision of the training manual included in Appendix A 
has attempted to clarify the points of confusion between these two index terms. 

The other area of confusion is major and deserves special comment. In the 
six training sessions it was pointed out that certain statements in the narra- 
tive text describe the job duties and the qualifications for the position that 
the ratee occupies rather than the ratee's actual qualifications for and per- 
formance in this position. When such a factual statement of the requirements 
for the position were included in the narrative text, it was not to be indexed 
since it told nothing about the ratee's qualifications and performance per se. 
Therefore, it was not considered to be an evaluative statement. Unambiguous 
examples of this type of narrative statement taken from the training material 
are the following: 

Example 1. As the Quality Control Chief he is responsible for the 
continuous updating of a number of SOP's as well as implement- 
ing the new ones that are required. 

Example 2. Chief XX is presently serving in an RM1 billet. This 
is because he made Chief Petty Officer in November 1970. 

However, when confronted with indexing the reliability study data base 
itself, this indexing convention was frequently misconstrued by the three re- 
liability indexers who had not had the extensive indexing experience that the 
experienced indexer had had. The following two examples illustrate where the 
confusion arose. 

Example 3. Ratee's assignment demands particularly delicate tact, 
due to his working among civilians upon whom he must depend 
for cooperation. 
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The experienced indexer did not assign any index terms to this sentence, con- 
cluding that it was a statement defining the requirements of the job position 
rather than how the ratee performed in the job. All of the other three relia- 
bility indexers inferred that the individual evaluating the ratee was actually 
referring to the fact that the ratee possessed tact in interfacing with the 
civilian community. The principal investigator used the terms CONDUCT, INTEG- 
RITY, AND PRIDE and REPRESENTATION to index this sentence, as did inexperienced 
indexer B. Inexperienced indexer A used only CONDUCT, INTEGRITY, AND -PRIDE. 
There was substantial agreement among the three less experienced reliability 
indexers, but none of them matched the indexing decision of the experienced in- 
dexer that they were trying to emulate. Another example of this type of dis- 
agreement is the following: 

Example 4. Additionally, he must supply satisfactory solutions to 
the many problems of the Company Commanders in connection with 
these services. 

Again the experienced indexer considered this comment to be a statement defin- 
ing the requirements of the job position whereas the other three reliability 
indexers attributed the skill required to the ratee. The principal investiga- 
tor and inexperienced indexer B called the skill CONTROLLING; inexperienced in- 
dexer A called it PLANNING . 

Disagreement in which the experienced indexer did not assign any index 
terms and the other three reliability indexers did assign one or more terms is 
very noticeable in studying the three agreement matrices in which the three 
less experienced reliability indexers are compared with the experienced index- 
ex . This type of disagreement also occurred in the three comparisons among the 
less experienced indexers, emphasizing the general confusion that existed in 
how to handle statements of the type shown in Examples 3 and 4 above. Addi- 
tional training aimed at clarifying this area cf confusion most likely would 
markedly reduce this type of disagreement and raise the magnitude of k. 

Analysis of the level of agreement among the four reliability indexers in 
assigning numerical weights to each index term selected, based on the modify- 
ing adjectives and adverbs, was performed differently than the analysis of the 
level of agreement in selecting the index terms themselves. Selection of the 
index terms in this reliability study constituted a nominal scale whereas as- 
signment of a numerical weight to each index terra selected was an indexing de- 
cision involving an ordinal scale. Therefore, more powerful agreement statis- 
tics could be employed, Since numerical weights on a scale from 1 to 5 (New 
-2 to New 3) were assigned to each index term selected, it was possible to com- 
pute a product moment correlation coefficient between each pair of reliability 
indexers. The new transformed weights were used in these computations since 
this ordinal scale provided a more justifiable way of measuring the situation 
in which one indexer did not select an index term but the other indexer did 
(see Table 9). 

In addition to computing these six product moment correlation coefficients, 
another agreement statistic, weighted kappa, was also calculated in order to 
determine if it agreed with the results of the correlational analysis. In 1968 
Cohen published another article generalizing the kappa statistic to the situa- 
tion in which disagreements of varying gravity can be weighted accordingly. 21 
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Application of weighted kappa to quantifying the level of agreement in psychi- 
atric diagnosis was also shown by Cohen and his colleagues. 2 

Weighted kappa is an agreement statistic corrected for chance agreement, 
to be used when different kinds of disagreement are to be differentially weight- 
ed in the agreement index. The desired weighting is accomplished by an a 
priori assignment of weights to the r by c cells of the agreement matrix, and 
must be done very carefully because the weights assigned are an integral part 
of how agreement is defined, and therefore, how it is measured with weighted 
kappa (k ). Table 41 shows the weighting algorithm that was used in computing 
k for assessing the level of agreement in assigning numerical weights, based 
on the modifying adjectives and adverbs, to the index terms selected in the 
reliability study. The first step in computing :c was to construct a 6 by 6 
agreement matrix between each pair of reliability indexers that encompassed 
all of the pairwise numerical weights that were assigned to index terms based 
on their modifying adjectives and adverbs. These numerical weights were tabu- 
lated in the agreement matrix across all 96 documents in the reliability study 
data base. Using the first row of Table 41 as an example, if Reliability In- 
dexer I and Reliability Indexer II both had assigned a numerical weight of 3 
to the index term that they had selected, it represented perfect agreement in 
their interpretation of the superlativeness of the adjective or adverb modify- 
ing the index term. Therefore, the 3,3 cell was given an a priori weight of 
zero in computing k since perfect agreement should receive no penalty. If 
one indexer had assigned a numerical weight of 3 to the index term selected 



TABLE 41 

THE WEIGHTING ALGORITHM USED IN COMPUTING WEIGHTED KAPPA 
FOR ASSESSING THE LEVEL OF AGREEMENT IN ASSIGNING NUMERICAL WEIGHTS 
TO THE INDEX TERMS SELECTED IN THE RELIABILITY STUDY 
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and the other indexer had assigned a numerical weight of 2, they only dis- 
agreed by one position on the ordinal scale, and therefore, the 3,2 cells were 
given an a priori weight of one in computing k , penalizing this mild dis- 
agreement only slightly. In the extreme case, if one indexer had assigned a 
numerical weight of 3 to the index term selected and the other indexer had 
assigned a numerical weight of -2, they disagreed by five positions on the 
ordinal scale, and therefore, the 3,-2 cells were given aa a priori weight of 
five in computing k , penalizing this extreme disagreement the maximum possibl 
This same logic was applied in determining the weights to be used iu computing 
k throughout the remainder of the matrix. All of the diagonal cells were giv 
en a weight of zero since in no case should perfect agreement be penalized. 
All cells immediately off the diagonal were penalized by a weight of one; 
those cells slightly farther off the diagonal were penalized by a weight of 
two; and so on out to a penalty weight of five for the case of worst disagree- 
ment . 



The formula for computing k is 



Zw . .p . . 

K = 1 - , 13 
o) T,w . .p . . 

IJ CIJ 



where w^. . = a priori weight in cell ij 

p . . = observed proportion in cell ij 
oij * L J 

p , . = chance proportion in cell ij 

CIJ 



The standard error i f k is equal to- 
co 



Zw . , 2 p . . - (Zw . ,p , .) 2 

K ojo y N(Enr. .p . >) 2 

v ij*cij j 



A significance test of k , that is, a test of H : Population \: - Observed 
k = 0, is accomplished fey evaluating the norma? curve deviate W 



z = 



Table 42 shows the results of the correlational analysis and the weighted 
kappa analysis for the six pairwise comparisons between the four reliability 
indexers in assigning numerical weights to each index term selected, based on 
the modifying adjectives and adverbs, for the entire reliability study data 
base. The results of the correlational analysis are shown first in Table 42. 
The best agreement in assigning numerical weights to each index term selected 
was obtained between the experienced indexer and inexperienced indexer A, a 
correlation coefficient of .80. The best agreement in selecting index terms 
.themselves was also achieved between this same pair of indexers (see Table 40) 



89 







CO 








CO 


Pi 








rH 










CO 


!>% 








£j 
d 


W 
M 








<3 


H 


ED 






<J 




H 










a 






<4 




w 






p-« 


H 


p 






P-I 


P 










H 


CO 












[-M 






<* 




CO 




p 


H 


§ 






r-ri 


P 


w 


pq 






W 


H 






p-H 


Pi 




<3 




LJ 


Pi 


X 






rH 


W 








p 








>J£ 


o 












H 






3-3 


w 




P 




H 


PC 


a 








H 


< 


CO 




p 




w 












£ 




<c 


W 


o 


H 


C\l 




w 


H 


M 


<r 


CO 






P 




H 




CO 


M 


w 


CO 


w 


H 


pq 




rH 


P3 


PC 






rJ 




o 




3 




c: 


H 


P 


H 








W 




< 


O 




Pi 






CO 










H 




w 




< 






pi 






% 


o 


H 




o 




H 


H 




H 


§ 


Pi 


23 




H 


O 


E3 


W 




< 


a 


f! 












W 




W 


w 




X 




Pi 






H 




Pi 


H 


O 






o 




Pi 




a 


pi 


H 


o 






H 


23 






w 


< 


O 








H 










CO 










CO 










< 






o 


CO 












55 






CO 


W 


M 






H 




















P 


Pi 








CO 








W 


o 








Pi 



















rH 




rH 




00 








CN] 










oo 








vO 












ro 






































N 


CO 








<N 




CO 




CO 




ro 










rH 




rH 




i — 1 




rH 




rH 




rH 




td 






























cu 






























qu 






























td 




o 














LO 








^« 




1*3 




3 


LO 




m 








vO 






























<r 












XJ 


b 




o 




o 




o 




o 




o 




o 




QJ 






























4J 




























































00 






























•H 






























QJ 










































































/— N 






















m 




CN 




00 


























00 




v£> 
















































































3 


m 




00 




O 




<r 




rH 




vO 










vO 












vO 




VO 




vO 






































4J 






























c c 






























QJ O 






























g ,H 






























O U 
































* 
































1 






o 








o 




<r 




ON 




4J QJ 






VO 




00 












vO 




vO 




CJ M 


1 


H 






t 




















3 M 






























XJ O 






























O O 






























M 






























P-i 




















































< 




pq 












CO 


U 


CO 




CO 




U 




n 




CO 










> 


O 


> 




> 




o 




o 


H 


> 












4J 










u 


QJ 


u 


QJ 






CO 




CO 


u 


rt 


u 


< 


u 


pq 


td 


X 


td 


X 




pq 


CJ 


O 


U 


QJ 


W) 


QJ 




QJ 




oo 


QJ 


oo 


QJ 






O 




QJ 


X 


•H 


X 


M 


X 


U 


•H 




•rl 




n 


U 


to 


u 


X 


QJ 


4J 


QJ 


QJ 


QJ 


QJ 


4J 


c 


u 




QJ 


QJ 


•rl 


•H 


QJ 




CO 




X 


T3 


X 


CO 


•rl 


CO 


•H 


X 


X 


U 


td 


XJ 


c 


QJ 


a 


QJ 


CJ 


QJ 


QJ 




QJ 




QJ 


QJ 


ctf 




C 


•H 


> 


•H 




•H 


TJ 


> 










TJ 


a 




H 




c 








C 




QJ 




a> 


c 


C 


s jc 




T3 


•H 


T3 


•H 


T3 


•H 


•rl 


U 




u 


•rl 


•H 


o 


u 




QJ 




QJ 




OJ 






C! 




cs 






o 


td 


U 


CJ 


H 


U 


TJ 


u 


TJ 


H 


QJ 


H 


QJ 


T3 


TJ 




W 


•rl 


C 


td 


a 


QJ 


C! 


QJ 


td 


•rl 


td 


•H 


QJ 


QJ 


0) 




H 


QJ 


p.. 


QJ 


U 


QJ 


O 






ex 




U 


CJ 


CO 


c 


•H 


•H 


•H 


•H 




•H 


c 


•rl 


QJ 


•rl 


QJ 


c 


c 


•rl 


QJ 




M 


U 


M 


QJ 


M 


QJ 


CJ 




CJ 




QJ 


QJ 




QJ 


■s 


QJ 


CJ 


QJ 


•H 


QJ 


•r( 


C3 


X 


C3 


X 


•H 


•rl 






•rl 


CU 


•rl 


a. 




D, 




•rl 


QJ 


•rl 


QJ 


M 


M 


•H 


-U 


H 


X 




X 


QJ 


X 


QJ 


U 


C 




c 


QJ 


QJ 


III 


QJ 


QJ 


QJ 


Cu 


QJ 


& 


QJ 




a. th 




•H 


P. 


CU 


PH 


pq 


Pi 




QJ 








X 










X 


X 








QJ 


QJ 


QJ 


QJ 


QJ 


QJ 




QJ 




QJ 


QJ 












CJ 


rC 


cs 


X. 


CO 


JC 


CO 


c 


c 








H 


4J 


H 


•H 


H 


•rl 


H 


> 


H 


> 


H 


•rl 



CJ 
QJ 
U 
QJ 



•rl 

rH 
4J 
C 

td 

u 

•rl 
•rl 

c 

00 



CO 
•H 



o 
o 
o 



o 

23 



td 



CO 

M u 

QJ CJ 

X QJ 

QJ H 

T3 QJ 

C CO 

•rl 

4-) 

JC o 

4J C 

o 

•H 

JC TJ 



•H M 

x: QJ 
S X 
QJ 

C TJ 
•rl C 
•H 

CO 

QJ TJ 
O QJ 



td 

4J 

CO 

c 

•H 



U 
C 
QJ 
•rl 

V-i 

QJ 

QJ x 

CO QJ 

o 

r& QJ 

rH JC 



4J 




M 




O 


I 




QJ 


o 


H 






QJ 


CO 




QJ 


■U 






H 


4J 


td 


td 


> 


JC 




i-i 


N 






CO 


QJ 


•rl 


JC 


CO 


4J 


QJ 




JC 


M-l 


4-) 


O 


o 




a. 


H 




H 


JC 


td 





QJ 3 

H C 

o 

U-i QJ 

QJ JC 

M 4J 
QJ 

JC M-) 

H O 

C 

• o 



o 




O 






li 


ec 












•rl 


•rl 


•r-j 


U 


QJ 




c 




TJ 


JC 


QJ 


C 


CO 




c 


c 




td 


U 


QJ 


td 




o 


•rl 


M 


42 




e 






u 




QJ 


o 


0 


QJ 






u 


CO 


X 


u 


u 


QJ 






td 


QJ 


QJ 


Cu 




U 








CJ 








bO 


o 




o 


c 


C 


MH 


td 


cn 


H 






td 


•H 


O 


QJ 








in 


St 




H 


H 


of 


M-l 






c 


QJ 


QJ 






O 




QJ 


•H 


V 


> 


C 


rH 










C 


QJ 


(S 


QJ 


4J 


•H 


td 


QJ 


QJ 


H 




> 




rH 


4J 


CO 


•rl 




4-J 


QJ 


QJ 


•rl 




o 


U 


H 


c 


H 


•H 


42 


CO 


4C 


QJ 


O 


td 




U 


td 


QJ 


4J 


a 


O 


u 


CJ 


•rl 




CO 




X 




•rl 


o 


M-l 


o 


QJ 


QJ 


QJ 




4-1 




M-J 






TJ 




QJ 


•rl 


td 


QJ 


o. 






CO 


4C 


c 


JC 


O 






rH 


CO 


u 


CO 




CJ 


M-l 


QJ 


u 


QJ 










O 


u 


X 


H 


u 




*j 


JC 




td 


a; 










o 


rH 


Cu 




M 




>, 


T3 


•H 


QJ 






QJ 




H 




4J 


& 


C 


"8 








XI 


td 




•rl 


td 


4J 


td 


00 


QJ 


1— 1 


H 






o 


CJ 


•rl 


QJ 


QJ 










•rl 




U 


u 


rH 






QJ 


4-1 




X 


u 


O 


O 


cu 




•rl 


QJ 


QJ 


o 


O 


4C 


4-) 


u 


c 


M 




CJ 




CO 


X 




00 
•H 


td 


Ot 


4J 


QJ 


3QJ 




CO 


QJ 


c 


(C 


rC 




TJ 


JC 




rH 






4J 




(C 




CO 


pC 


10 


I 






•H 


e 


•rl 




QJ 




4J 


O 








4-J 


o 


0 


td 




c 


QJ 


a> 




TJ 




CO 


td 


4J 


eg 


CO 






o 


QJ 








•H 


3 


u 


u 






X 


CO 








QJ 


H 


QJ 


QJ 








T3 


N 


td 


4J 


TJ 


4H 




TJ 


O 


0 


> 


U 


(C 


O 


tc 


QJ 


M 




QJ 


•rl 




•rl 


C! 




O 


QJ 


rH 




N 




•rl 




u 


4C 


QJ 


CJ 




TJ 


cd 


< 




H 


CO 


td 


<J 


QJ 


4J 



* 



ERLC 



90 



These findings corroborate each other in suggesting that an individual without 
a research background in only six training sessions can be taught not only how 
to select the most appropriate index terms but also how to consistently assign 
weights to these terms based on the modifying adjectives and adverbs . me 
other correlations reported in Table 42 are lower, but none is less than .64. 
All six correlation coefficients are significantly different from zero well 
beyond the^. 001 level of probability. 

On th-^ weighted kappa side of Table 42 the first column shows the value 
of k ; the second column shows the standard error of k ; and the last column 
lists the normal deviate, z, obtained by dividing k by its standard error. 
All of the z values are large, and consequently, highly significant, indicat- 
ing that in all six comparisons the null hypothesis that the obtained k does 
not exceed t^he chance level of agreement can be rejected. As was expected, 
the k valu§s are similar in magnitude to ir correlation coefficient coun- 
terparts. Again, the best agreement as measured by weighted kappa was ob- 
tained between the experienced indexer and inexperienced indexer A, a of 
.78. The three values shown in parentheses after the first three k t s 
listed in Table 42 were computed in order to determine the level of agreement 
achieved if those instances were excluded where the experienced indexer did 
not select an index term, and consequently, did not assign a numerical weight 
but the other less experienced indexer did select an index term and assigned 
a weight to it. This proved to be the area of major confusion in executing 
the reliability study as pointed out earlier in this section in discussing 
the results of the kappa analysis of level of agreement in selecting the index 
terms themselves. Instances where the experienced indexer did not assign a 
weight but the other indexer did form one row in the weighted kappa computa- 
tional matrix. This row can be omitted from the computation, resulting in a 
value for k that ignores this major area of confusion and takes into account 
only those instances where both indexers selected an index term, and conse- 
quently, assigned a weight. The gain in the value of k is not very large for 
the comparison between the experienced indexer and xi_3x, arienced indexer A 
when k was recomputed in this fashion. However, the gain was quite substan- 
tial in the other comparisons between the experienced indexer and the princi- 
pal investigator and between the experienced indexer and inexperienced indexer 
B. This suggests that with additional training to clarify this area of con- 
fusion and with more indexing experience, the level of agreement among the 
four reliability indexers could possibly be raised to a value of .80 to .85 
as measured by any of the three agreement statistics employed in this study. 
However, values in the .90 T s are the ultimate objective. 

In conclusion, it might be of interest to point out that the initial ex- 
pectation in beginning this reliability study was that it would be extremely 
difficult to train nonresearch-oriented individuals to consistently index the 
narrative sections of Evaluation Report forms using the complex content analy- 
sis methodology that had been developed in the pilot study. The surprising 
result is that in only six training sessions a quite respectable level of agree- 
ment was achieved. Moreover, one of the inexperienced indexers showed a higher 
level of agreement with the experienced indexer than the principal investigator 
did, and the other inexperienced indexer agreed with the experienced indexer 
almost as well as the principal investigator. The intuitive feeling that the 
reliability indexers had after completing the reliability study was that the 
most difficult part of learning to index consistently was over and that with 
additional practice and some review training sessions they could improve their 
indexing skill. 
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SECTION 6. FUTURE AREAS OF INVESTIGATION 

In the subsequent 10-month period of this research project beginning 
March 1, 1973 and concluding December 31, 1973, additional studies of the 225 
Evaluation Reports in the pilot study sample and the 444 Evaluation Reports in 
the cross validation and generalization samples will b ~ried out in order 
to devise valid, short-cut methods of indexing the nar. :e content of these 
reports based on the more lengthy and complex content analysis methodology 
that has already been developed. The issues of trainability and reliability 
in indexing this type of narrative text will be studied further. The follow- 
ing specific tasks are being undertaken: 

A. Short-cut Indexing Methods 

Efforts are in progress to develop valid, short-cut methods for index 
ing the narrative content of Evaluation Reports by capitalizing on the finding 
resulting from the various stepwise discriminant analyses that have been per- 
formed. The variables that are entered into the discriminant function at the 
first five to ten steps in the analysis appear to be the key variables in dis- 
criminating among the three criterion groups. These variables then form one 
target for study, i.e. , how to extract this differentiating information from 
the narrative text in a simple but reliable fashion that will achieve as good 
or nearly as good classification accuracy as the longer, more complex index- 
ing methodology. This approach is being used to develop optimum streamlined 
classification algorithms for all four ratings represented in the research 
data base studied thus far, i.e., AT's, BT ! s, CS's, and RM T s. 

In addition, the literature is being searched for work that may be 
relevant to the objective of developing valid, short-cut methods for indexing 
the narrative content of Evaluation Reports. 

B. Extension of the Inter-indexer Reliability Study 

An extension of the reliability study is being conducted using four 
reliability indexers. Two of the original reliability indexers (inexperienced 
indexers A and B) , after refresher training using the updated version of the 
training manual, will independently index a different set of 48 Evaluation Re- 
ports than that used in the original reliability study and their indexing de- 
cisions will be compared to the judgments of the experienced indexer. The 
level of agreement between each of the inexperienced indexers and the experi- 
enced indexer after a second exposure to reliability indexing will be calcu- 
lated to determine if the level of agreement can be improved with additional 
training and experience. In addition, two new reliability indexers will be 
trained 5 and they will independently index the original set of 48 Evaluation 
Reports. Their indexing decisions will be compared to those of the experienc- 
ed indexer to determine if the level of agreement achieved is comparable to 
that found in the first reliability stt^y. 
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PURPOSE 

The purpose of this manual is to train nonresearchers in the content 
analysis techniques developed in a personnel research study to analyze the 
narrative sections of Navy performance evaluations for senior enlisted person- 
nel in pay grades E-7 (Chief Petty Officer), E-8 (Senior Chief Petty Officer), 
and E-9 (Master Chief Petty Officer), 1 The objective of this study was to 
provide . personnel decision makers (e.g., selection boards and detailers) with 
a standardized way of detecting valid and discriminating indicators of on-job 
performance in narrative evaluation comments, 

BACKGROUND 

Section 19, Evaluation Section, of Evaluation Report Form NAVPERS 1616/8* 
is designed to permit the rater (evaluator) to compare the ratee with all 
others of his rate known to the rater on 13 specific aspects of on- job perform 
ance as well as to make an overall comparative evaluation of the ratee T s per- 
formance (see Figure A-l) . Ratings are made by marking the column of tie rat- 
ing distribution into which the rater evaluates that the ratee falls (f c r ex- 
ample, top 1% for superlative performance). Section 19R of this form provides 
space for the rater to write narrative evaluation comments to describe further 
the ratee 1 s performance and qualifications. Section 19S of this form provides 
space for the rater to write narrative justification comments and is required 
to support any marks assigned to the top or bottom 10, 5, or 1% columns of 
Section 19. 

Sections 19R and 19S are referred to as the narrative text of the Evalua- 
tion Report since they are the only portions of the report where the rater 
uses his own words to assess the on-job performance of the senior enlisted man 
that he is rating. Thus far the narrative evaluation and justification sec- 
tions of the Evaluation Report have not been exploited systematically in mak- 
ing personnel decisions because narrative text tends to resist objective analy 
sis and interpretation. However, results from a content analysis study of the 
narrative text 1 strongly suggest that there are stable differences among the 
performance characteristics of chief petty officers that are reflected in the 
narrative statements written about them by evaluators. Furthermore, these 
differences are both identifiable and quantifiable. The remainder of this 
manual presents a set of explicit and detailed guidelines for identifying, in- 
dexing or labeling, and quantifying (by means of a weighting scale) the con- 
cepts and ideas represented in the narrative text of Navy performance evalua- 
tions for senior enlisted personnel. These quantified labels have been shown 
to discriminate or differentiate between superior ratees and their slightly 
less qualified colleagues. 



This form subsequently has been replaced by another form that can be scanned 
by an optical character reader; however, the content of the two forms is 
essentially the same. 
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17. BiLLET LEVEL ASSIGNED: 

© Above Preient Pay G'ode all Above or Below 

O a . D o r , explain m Block 19R) 

At P'eient Pay Grode 

Q Below Pr?i«nt Pay Grode 



18. PAY GRAD[ 

E8 O 
E9 Q 



0. TREND OF PERFORMANCE DURING THIS PERIOD: 

Declining Q .Improving Q Steady Q 

Incontinent Firit leport for rotot 



P. YOUR ATTITUDE TOWARD HAVING RATEE CONTINUE 
UNDER YOUR COMMAND: 

Prefer not to hove ^) 

Pleaied to have J© 



Willing to hove © 

Porttculorly desire to hove ^) 



0. DURING THIS PERIOD HAS RATEE BEEN: YES NO 

( ]) Individuall y reported on in ony commendotory wOy ' ^) ^) 

(21 The subject of disciplinary action (either military © © 

or civilion)? If YES, explain 



19R, EVALUATION COMMENTS: Use lor information specilied elsewhere ond to describe further roteo'i performance and 
quolif tc otions . (Required (or some marks m Blocks II, 17, ond 190.) 



(If n«cesiOty, cominue on otiached sheet) 



* 19S. JUSTIFICATION COMMENTS: Use only to document ony Top Bottom 10/5-1% Morki in Section 1 9A thru 19N. 



(if necessary, continue on attached sh e e''i 



20. Thu report muit be signed by »he Initiot.ng OffiCiol ond the Reviewinq Ofliciol 



A. In.tiotmg Ojf<c<ol 



B. Reviewing Ofhcol 



Figure A-l. Evaluation Report Form NAVPERS 1616/8 (a 75 percent 
photo reduction of the original form) 
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RATEES AS MANAGERS 

Senior enlisted personnel in pay grades E-7/8/9 are all managers in the 
sense that they all are responsible for the supervision of other enlisted men 
whose work they direct. Therefore, the unifying focus in this manual is on 
the assessment of a 'chief petty officer (ratee) as a manager. The operations 
of a manager may differ from one organization or from one institutional set- 
ting to another; however, the functions of a manager are common to all. The 
task of the manager is one of selecting goals and designing and maintaining 
an environment that makes possible the performance of individuals working to- 
gether in a group to attain these goals. Chief petty officers in pay grades 
E-7/8/9 are junior level managers, and as such, they must perform technical as 
well as managerial functions. Table A-l shows a hierarchy of 29 index terms 
or labels that can be used to characterize the on- job managerial performance 
of chief petty officers. These index terms are the terms to be imposed onto 
the narrative text to give it objective structure and to systematize the way 
that this text is analyzed and interpreted. Jote in Table A-l that the 29 
index terms are divided into three sections. The firs!; section contains seven 
specific MANAGEMENT FUNCTIONS that many authorities on management practice 
agree are the characteristic duties of all managers . 2 ' ^ ,t+ ' 5 ' 6 Although some 
authorities believe that there are more, less, or different functions perform- 
ed by managers, these seven functions were selected because they are repre- 
sentative of the duties that chief petty officers actually perform. 

The second section of Table A-l contains index terms for 13 specific 
SKILLS AND ABILITIES considered to be important by Navy supervisory personnel 
in performing effectively as a chief petty officer. While some authorities on 
management practice consider making a judgment about whether or not an indi- 
vidual possesses a skill, quality, or ability to be a subjective process, Navy 
evaluators do repeatedly call out these specific qualities in their narrative 
evaluations because many of these qualities are dimensions on which they rate 
the ratee in Section 19 of the Evaluation Report. The first section of Table 

A-l MANAGEMENT FUNCTIONS deals with how a ratee performs his managerial 

functions and is result oriented, while the second section — -SKILLS AND ABILI- 
TIES contains index terms that relate to an individual's characteristics and 

qualities which, if used, may help him achieve good results. 

The third section of Table A-l PRODUCTIVITY AND ACHIEVEMENT is the 

most result-oriented section of the hierarchy. Here are included the measures 
of overall performance. DRIVE and SERVICE MOTIVATION (a specific type of 
drive) are included in this section since drive is considered to contribute to 
successful performance. POTENTIAL also is included here since potential is a 
measure of future performance. AWARDS AND PUNISHMENT, REPUTE, and ASSET TO 
THE NAVY represent acknowledgments of an individual's performance, either 
positive or negative acknowledgment. 

QUANTIFYING THE INDEX TERMS 

It is not enough to simply label a narrative statement with the most ap- 
propriate index term since the statement may have been a highly positive, 
quite positive, neutral, quite negative, or highly negative one. For example, 
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TABLE A-l 
HIERARCHY OF INDEX TERMS 

MANAGEMENT FUNCTIONS 
CONTROLLING 

LEADERSHIP AND DIRECTING 

ORGANIZATION 

PLANNING 

REPRESENTATION 

STAFFING 

USE OF COMMUNICATION 

SKILLS AND ABILITIES 
COMMUNICATION 

CONDUCT, INTEGRITY, AND PRIDE 

COOPERATION 

ENDURANCE 

FLEXIBILITY 

GROOMING AND ATTIP.E 

INITIATIVE 

INTELLECTUAL FUNCTIONING 

PROFESSIONALISM 

RELIABILITY AND DEPENDABILITY 

RESOURCEFULNESS 

RESPONSIVENESS 

TECHNICAL SKILLS 

PRODUCTIVITY AND ACHIEVEMENT 
AWARDS AND PUNISHMENT 
DRIVE 

SERVICE MOTIVATION 
POTENTIAL 
REPUTE 

ASSET TO THE NAVY 
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in order to differentiate between the ratee who plans superbly and the ratee 
who plans inadequately, a weighting scale was devised to be applied to each 
index term that is used (see Table A-2). The weighting scale contains five 
numerical values ranging from 5, the positive end of the scale, to 1, the 
negative end of the scale. Under each numerical value in Table A-2 there are 
listed examples of adjectives or adverbs that may be used by the rater to de- 
scribe a ratee' s performance. These lists of words provide clues to the in- 
dexer as to which numerical value to assign to an index term. As a simple ex- 
ample, if the rater commented that the ratee was highly cooperative, this 
statement would be indexed as COOPERATION and assigned a weight of 4 since 
highly is listed as an example under numeral 4 in Table A-2. 

However, in many cases the indexer will have to exercise his own judgment. 
Some of the words that will require such a judgment are the following: . abnor- 
mally , absolutely , all, always, consistently , constantly, fully, immeasurable, 
intense, no doubt, obvious, totally, unquestionably, and uppermost . An index- 
er will encounter many more words than those enumerated above that will require 
good judgment in choosing the most appropriate weight to use. For example, 
consider the following statement: "He is always resourceful.' 1 This statement 
would be indexed as RESOURCEFULNESS 4, because the statement could have been 
worded in a stronger way. The rater could have said, "He is always fully re- 
sourceful.' 1 This stronger statement would have been indexed as RESOURCEFUL- 
NESS 5. The words always and fully in most cases indicate the use of a 4 on 
the weighting scale; yet in the above example where they appear together, the 
statement is made so strongly that a weight of 5 is the correct indexing de- 
cision. However, there 1 are other 4-weighted modifiers that when used in com- 
bination still remain a 4, for example, remarkably well or highly effective, 
because the modifying phrase could have been worded even more strongly, for 
example, extremely well or most effective. It is important to remember when 
using the weighting scale that the indexer should ask himself, "Could this 
statement have been phrased in another wiy by the rater that would have made 
it a stronger or a weaker statement?" The i T taxing convention to be followed 
for the modifier quite is to consider it to have a positive connotation and 
to give it a weight of 4 as in the following example: "He has been quite 
resourceful in making do with available parts." RESOURCEFULNESS 4. However, 
there may be instances in which the indexer would assign a 2 weight to the 
modifier quite if the context was sufficiently negative t:6 warrant it as in 
the following example: "His tendency to be quite overweight detracts from his 
overall appearance." GROOMING AND ATTIRE 2. Note that AWARDS AND PUNISHMENT 
is given either a 5 or a 1 weight since there is no degree of variance. Either 
the ratee was given an award or not, or was punished (disciplined) or not. 



SPECIAL INDEXING CONSIDERATIONS . 

An alphabetical dictionary of the 29 index terms appears at the end of 
this discussion. For each term in the dictionary, a definition is given, ex- 
amples of narrative text indexed with the term are cited, and usage rules to 
guide the indexer in choosing this term or another term are supplied. Careful 
study of the dictionary will instruct the new indexer in how index terms and 
their numerical weights should be assigned in order to ensure a systematic and 
objective application of the indexing procedures explained in this manual. 
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excellent 



4 

good 



TABLE A- 2 
WEIGHTING SCALE 

3 

1 average | 



2 

P° or 



1 

poorest 



superlative 
best 



comparative 

better than 
most 



average 



comparative 

not as good 
as most 



superlative 
worst 



above 

reproach 

beyond 
reproach 

boundless 

exceptional 

extra- 
ordinary 

extremely 

finest 

flawless 

greatest 

highest 

ideal 

little to be 

desired 
limitless 
maximum 
most 
never 

outstanding 

paramount 

perfect 

profound 

sterling 

superb 

superior 

surpassed by 

none 
top/topnotch 
unimp pachab 1 e 
unique 
unlimited 
unmatched 
utmost 

without equal 
without 

exception 
100% 



above average 

better 

commendable 

complete 

deep 

definitely 

easily 

effective 

efficient 

eloquent 

eminent 

exceeds 

excels 

exemplary 

expeditious 

experienced 

expertise 

extensive 

favorable 

great 

high/highly 

immaculate 

immensely ' 

impeccable 

impressive 

innate 

inspires 

instills 

invaluable 

keen 

laudable 

leading 

marked 

meticulously 

model 

much 

noteworthy 

particularly 

rapidly 



EXAMPLES 

adequate 
aptly 
capable 
competent 
generally 
moderate 
satisfac- 
tory 
sufficient- 

iy 

usually 



4-good (Com:. ) 
rare 

remarkable 

significantly 

skillful 

smoothly 

solid 

strongly 

surpassed 

thorough 

tremendous 

truly 

unstinting 
valuable 
vast 
very 



declining 
quality 
deficiency 
detrimental 
fair 

in need of 

insufficient 

lack of 

lower iihan 
average 

lowering of 

negatively 

spotty 

unfortunate 

unwisely 

weak in 

with the ex- 
ception of 



bottom 

least 

lowest 



NOTE: AWARDS AND PUNISHMENT is assigned a weight of either 5 or 1. 
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Although some indexing examples may not always seem logical to the new indexer, 
each indexing decision has .been meticulously and thoroughly considered. The 
examples presented in the alphabetic dictionary represent a distillation of 
two years 1 of indexing experience and constitute a self -ins truct ional compila- 
tion of crucial indexing rules and conventions that the new indexer needs to 
know in depth in order to be able to index the narrative text of Evaluation 
Reports accurately and consistently. 

There are several indexing considerations that should be kept in mind as 
they will assist the indexer in maintaining consistency and will help resolve 
indexing dilemmas. Sections 4A and 4B of Evaluation Report Form NAVPERS 1616/8 
provide a description of the ratee' s primary and collateral duties. These sec- 
tions should be read as background information before beginning to index uhe 
narrative text. An example of these two sections is shown below: 

4A. DESCRIPTION OF PRIMARY DUTIES DURING THIS REPORTING PERIOD 

Ratee is assigned as supervisor of Intermediate Level Mainte- 
nance on communications, navigation, and radar systems in- 
r stalled in the EA-3B, EP-3B, and EC-121M aircraft. 

4B. DESCRIPTION OF COLLATERAL DUTIES AND/OR SPECIAL ASSIGNMENTS 
DURING THIS REPORTING PERIOD 

Ratee is assigned on a rotational basis as the Avionics Super- 
visor at the squadron detachment in Danang, RVN. He also 
stands Assistant Squadron Duty Officer watches. 

Information about a ratee' s job duties is needed when indexing because a 
statement about an individual's achievement may not be in terms of performing 
a managerial function but rather in terms of his overall performance. Consid- 
er the following statement from the narrative text: "Ratee 's supervision of 
maintenance of the communication and radar systems has been outstanding." 
This statement would be indexed as PRODUCTIVITY AND ACHIEVEMENT 5 because it 
is an assessment of the ratee 's overall job performance as a supervisor of 
intermediate-level maintenance rather than performance of the controlling func- 
tion of maintaining equipment in order to assure accomplishment of plans. 

Another consideration to keep in mind is that when a description of a job 
or job duties is included in the narrative text, this description is not in- 
dexed since it is a factual statement describing the qualifications needed to 
perform a specific job or the duties of that job. Therefore, the statement is 
about the job itself and not about the ratee. Even if such a statement is 
modified by adjectives or adverbs, it still is not indexed if it refers to how 
a job should be performed and not to how the ratee actually performs a job. 
As an illustration of this convention, if planning is mentioned as one of the 
ratee 's duties, this is a factual statement since it is the duty of all manag- 
ers to plan and no index term would be assigned to this statement. For example, 
"Chief XX is required to develop procedural methods of accomplishing the divi- 
sion workload." Even if an adjective or adverb is added to this statement 

"Chief XX is required to develop effective and efficient procedural methods of 
accomplishing the division workload," the statement is still about a specific 
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job duty or requirement and the modifiers refer to how the job should be per- 
formed. The modifiers do not refer to the ratee per se and, therefore, this 
statement should not be indexed since it is not evaluating the ratee. However, 
if a statement is a qualitative statement and refers to the ratee, then it be- 
comes a statement of evaluation and is indexed. If the rater said that the 
ratee plans well on the job, then a value judgment has been rendered about what 
kind of a planner the ratee is. For example, TI Chief XX has developed effective 
and efficient procedural methods of accomplishing the division workload"; this 
statement would be labeled PLANNING 4. The statement no longer is a factual one 
but has become an evaluative comment about the ratee rather than a statement de- 
scribing a job requirement, "Chief XX has developed procedv cal methods of accom- 
plishing the division workload," would be labeled PLANNING 3 since the rater 
thought it worthwhile to mention this information and the statement, therefore, 
evaluates the ratee although no modifying adjective or adverb was used. Always 
keep in mind that indexing decisions are made in terms of the qualifications 
that a ratee actually possesses that can aid him in performing a job, in terms 
of how a job is performed by a ratee , or in terms of the results achieved* 

Also keep in mind that in deciding on which numerical weight to use, modi- 
fying adjectives and/or adverbs must be associated with the idea or concept be- 
ing indexed and not with another idea or concept in the same sentence. For 
example, consider the following statement: "His resourcefulness in completing 
his tasks in the most efficient and thorough manner is noted." Most is associ- 
ated with the manner in which the ratee performs his tasks and, therefore, quali- 
fies PRODUCTIVITY AND ACHIEVEMENT as a 5- The ratee f s resourcefulness is not 
modified but, it is stated that he possesses that characteristic. It helped to 
make the 5 weight possible for PRODUCTIVITY AND ACHIEVEMENT, but the first part 
of the statement is only indexed as RESOURCEFULNESS 3. To be given a 5 weight, 
the statement would have had to have been, "His outstanding resourcefulness in 
completing his tasks in the most efficient and thorough manner is noted." When 
there is no modifier given for an evaluative statement, a 3 weight is assigned 
to the index term selected (e.g., "His planning efforts have led to..." would 
be labeled PLANNING 3). Also, if a modifying adjective or adverb that falls 
at the 3 position on the weighting scale is included in the evaluative state- 
ment, the index term selected still would be given a weight of 3 (e.g., "His 
competent planning has led to..." would also be labeled PLANNING 3). Each 
statement indexed has to be regarded as a separate entity lest confusion and 
inconsistency result. Always remember the rule: How else could this statement 
have been worded? 

However, if an adjective occurs before a string of words and phrases, 
this adjective modifies each word or phrase in the string until there is a 
clear break in the sentence structure, or until the adjective could not logi- 
cally and/or grammatically be associated with a particular word or phrase. 
For example, "His outstanding technical knowledge and organizational ability 
ha^e contributed to..." would be indexed as TECHNICAL SKILLS 5 and ORGANIZA- 
TION 5. The adjective outstanding modifies both phrases. Note that organiza- 
tion in the above example is referred to as a skill; yet it is placed under 
MANAGEMENT FUNCTIONS in the hierarchy of index terms. It is often difficult 
to differentiate between the performance of c. function and the function as an 
actual skill that an individual may have. For example, there is a definite 
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ability to lead or skill of leadership; yet it also is a very important func- 
tion performed by managers . These subtleties in word meaning and usage are 
part of the expressive fabric of the English language and continue to plague 
those who strive to achieve precision in systematizing the information content 
of writcen discourse. At some point arbitrary rules have to be imposed. In 
this content analysis scheme, both organization and leadership are considered 
to be principally management functions ratneF than skills an individual may 
possess. 

Ic is also important to remember when indexing and applying the weighting 
scale to be careful to take note of the words that are actually contained in 
the definition of an index term. Consider the definition of COMMUNICATION; 
COMMUNICATION refers to the expression of thoughts and feelings through the 
spoken or written word in a clear and concise manner. Therefore, the indexer 
should regard the ability to communicate in a clear and concise manner as an 
average ability and index any statement phrased similarly as COMMUNICATION 3. 
If a ratee is said to communicate in a very concise manner, then this state- 
ment would be indexed as COMMUNICATION 4. 

Every attempt has been made to present the information contained in this 
manual in as explicit and lucid a form as possible. However, indexing remains 
more of an art than a science for all of the reasons alluded to previously. 
As an indexer you will encounter segments of narrative text for which only 
your considered judgment can help you arrive at the final decision. It is im- 
portant, though, that you try to keep your judgments as consistent as possible. 
The best way to assure consistency is to keep records of difficult or marginal 
decisions and, if possible, of the basis on which these decisions were made. 
Table A-3 presents a glossary of indexing decisions that were made by one ex- 
perienced indexer to handle the appearance of ambiguous or troublesome words 
and phrases in narrative text. Use this table as an extra indexing guide. 

It is recommended that the new indexer become thoroughly familiar with 
this training manual before attempting to index the narrative sections of 
Evaluation Reports. If he can compare his independent trial indexing deci- 
sions with those o. an experienced indexer, this procedure will serve to pin- 
point areas of confusion in his understanding of the indexing rules and con- 
ventions. Frequent rereading of the manual will help to guarantee that the 
rules are applied the same way from day to day. 
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TABLE A-3 



GLOSSARY OF WORD CLUES TO THE USE OF INDEX TERMS 



Acts with ease = CONDUCT, INTEGRITY, AND PRIDE 
Aggressive = DRIVE 
Agreeable = COOPERATION \ 

Alert = INTELLECTUAL FUNCTIONING (See Also RESPONSIVENESS) 
Amiable/Affable = COOPERATION 

Appraisal (assessment) of personnel = STAFFING 
Background in rate = TECHNICAL SKILLS 
Bearing = CONDUCT, INTEGRITY, AND PRIDE 
Can-do attitude = DRIVE 
Common sense = INTELLECTUAL FUNCTIONING 
Congenial = COOPERATION 

Constant = RELIABILITY AND DEPENDABILITY 
Coordinate = ORGANIZATION 
Decisiveness = SKILLS AND ABILITIES* 
Delegate = ORGANIZATION 
Deliberate = SKILLS AND ABILITIES* 
Demeanor = CONDUCT, INTEGRITY, AND PRIDE 
Deportment = CONDUCT, INTEGRITY, AND PRIDE 

Devoted (dedicated) to duty or the Navy = SERVICE MOTIVATION 
Devotes (dedicates) long hours = DRIVE 
Diligent = DRIVE 

Disposition = CONDUCT, INTEGRITY, AND PRIDE 

Dynamic = DRIVE 

(his) endeavor = DRIVE 

Expediently = RESPONSIVENESS 

Expeditious = RESPONSIVENESS 

Expertise = TECHNICAL SKILLS 

Forceful = DRIVE. 

Forcefulness of expression = COMMUNICATION 

Foresight = SKILLS AND ABILITIES 

Friendly = CONDUCT, INTEGRITY, AND PRIDE 

Genial = CONDUCT, INTEGRITY, AND PRIDE 

Humor = CONDUCT, INTEGRITY, AND PRIDE 

Imagination = RESOURCEFULNESS 

Ingenuity = RESOURCEFULNESS 

Innovative = RESOURCEFULNESS 

(has) instituted = INITIATIVE 

Inventory = PLANNING + CONTROLLING 

Judgment = INTELLECTUAL FUNCTIONING 

Logical (keen) mind = INTELLECTUAL FUNCTIONING 

Mature = RELIABILITY AND DEPENDABILITY 

New ideas = INITIATIVE 



If ratee is decisive (deliberate) in his thinking, index 
as SKILLS AND ABILITIES, If ratee is decisive (deliberate) 
in his behavior, index as CONDUCT, INTEGRITY, AND PRIDE. 



TABLE A-3 (CONT.) 



GLOSSARY OF WORD CLUES TO THE USE OF INDEX TERMS 



Original thinking = INITIATIVE 

Perseverance = ENDURANCE 

Recommended changes = INITIATIVE 

Sincere = CONDUCT,' INTEGRITY, AND PRIDE 

Stable = RELIABILITY AND DEPENDABILITY 

Suggestion making = INITIATIVE 

Supervising men * LEADERSHIP AND DIRECTING 

Supervising tasks = MANAGEMENT FUNCTIONS 

Tactful (used alone) = CONDUCT, INTEGRITY, AND PRIDE 

Tactful with men or subordinates = LEADERSHIP AND DIRECTING 

Talent = SKILLS AND ABILITIES 

Traffic flow pattern = PLANNING + CONTROLLING 

Versatile = SKILLS AND ABILITIES 

Vigor = DRIVE 

Zeal = DRIVE 



ALPHABETICAL DICTIONARY OF INDEX TERMS 



Format: The name of each index term is shown 
in all capital letters at the beginning of the 
definition of the term. The definition is followed 
by a SEE ALSO entry if there is one. The last sec- 
tion of the format presents a number of examples of 
how each term in the dictionary was used. The 
number following each index term is the weight 
assigned to it by the indexer. Additional indexing 
clues may also be given. 
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ASSET TO THE NAVY refers to the worth or value of having an individual contin- 
ue his career in the Navy. 

EXAMPLES : 



Ratee has been an exceptionally 
fine asset to this command. 



ASSET TO THE NAVY 5 



He has been a great asset to 
DESCOL and the Navy. 

Ratee has proven to be a definite 
asset with his outstanding pro- 
fessional and instructional ability 
in a relatively short time. 



ASSET TO THE NAVY 4 



ASSET TO THE NAVY 4 
PROFESSIONALISM 5 
COMMUNICATION 5 



He is an asset to the ship and 
the naval service. 

J 


ASSET 


TO 


THE 


NAVY 


3 


Additional Indexing Clues: 












One of the most outstanding men in 
the Navy today. 


ASSET 


TO 


THE 


NAVY 


5 


He is an outstanding military man. 


ASSET 


TO 


THE 


NAVY 


5 


Valuable asset to the service 


ASSET 


TO 


THE 


NAVY 


4 


Great value to the U.S. Navy 


ASSET 


TO 


THE 


NAVY 


4 


Asset to this command 


ASSET 


TO 


THE 


NAVY 


3 


Valuable to any command 


ASSET 


TO 


THE 


NAVY 


3 


Credit to the Navy 


ASSET 


TO 


THE 


NAVY 


3 


Credit to the squadron 


ASSET 


TO 


THE 


NAVY 


3 


Asset to the command of the Navy 


ASSET 


TO 


THE 


NAVY 


3 



ASSET TO THE NAVY is to be used only when the individual himself is con- 
sidered to be an asset and not one of his traits. If one of his traits 
is an asset to the Navy, use asset as a qualifier for the trait* How- 
ever, if his trait is an asset in performing a specific function, asset 
then qualifies the index term assigned to the function. For example, 
"His cooperative nature is a great asset to the Navy." COOPERATION 4. 
However, "His cooperative nature is a great asset in the performance of 
his coordinating responsibilities with Facilities Control and the Re- 
ceiver Site/ 1 would be indexed as COOPERATION 3; ORGANIZATION 4. 
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AWARDS AND PUNISHMENT are measurements of an individual's job performance as 
reflected in his receiving official commendatory recognition or penalty, 
or his being recommended for such an award or penalty. 



EXAMPLES: 

BTC was recommended for an achieve- 
ment medal for his leadership 
efforts during the 1969 WESTPAC 
Cruise . 

Awarded the Navy Commendation 
Medal for meritorious service 
while attached to Fighter Squadron 
ONE FIVE ONE embarked in USS CORAL 
SEA (CVA-43) during combat opera- 
tions from 10 October 1968 to 30 
March 1969. 

He was awarded an oral admonition. 
Additional Indexing Clues : 
Reported in a commendatory way 
Relieved of duties 



AWARDS AND PUNISHMENT 5 



AWARDS AND PUNISHMENT 5 



AWARDS AND PUNISHMENT 1 

AWARDS AND PUNISHMENT 5 
AWARDS AND PUinISVMENT 1 



Use AWARDS AND PUNISHMENT only once no matter how many awards a ratee 
received if they are all mentioned in one statement. For example, 
"He won five awards for outstanding performances last year," would be 
indexed as AWARDS AND PUNISHMENT 5. 

Use this term even if the award is given to a ratee 's squadron, command, 
or other such unit. For example, "He contributed directly to the annual 
OP-EVAL award of 'Outstanding' given this division by th DCA Inspection 
Team during their recent visit to this Command." AWARDS AND PUNISHMENT 5. 

Do not index that part of the sentence following the award statement 
which tells about the performance for which the ratee won the award. 
This is all considered to be part of the award statement. For example, 
"Ratee was recommended for and received the Navy Achievement Medal for 
his outstanding professional performance during this reporting period," 
would be indexed simply as AWARDS AND PUNISHMENT 5. 



NOTE: AWARDS AND PUNISHMENT is assigned a weight of either 5 or 1. 
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COMMUNICATION refers to the expression of thoughts and feelings through the 
spoken or written word in a clear and concise manner. Includes public 
speaking, reports , records , and letters . 

The ability to communicate is reflected in the choice of words or vocabu- 
lary, the organization of the expressed thoughts, phrasing, sentence 
structure, paragraphing, and overall clarity and forcefulness of expres- 
sion . 

SEE ALSO: USE OF COMMUNICATION 
EXAMPLES : 



His use of the English language is 
excellent and he expresses himself 
exceptionally well, both orally and 
in writing. 

Ratee's ability to correctly speak 
the English language is outstanding. 

He utilizes a well-rounded vocabulary 
to very effectively express himself. 

He possesses an average command of 
the English language both orally 
and in writing. 

He is capable of expressing himself 
clearly and adequately. 

He is soft spoken but speaks well; 
however, his written work, particu- 
larly spelling, could use improve- 
ment. 



COMMUNICATION 5 
COMMUNICATION 5 



COMMUNICATION 5 



COMMUNICATION 4 



COMMUNICATION 3 



COMMUNICATION 3 



COMMUNICATION 4 
COMMUNICATION 2 



Additional Indexing Clues : 

Excellent knowledge of English 
language 

Command of language superb 

Grammar excellent 

Vocabulary excellent 

His reports are accurate. 

Expresses well in writing 

Expresses well orally 

Expresses well orally and in writing 

Speaks with ease 



COMMUNICATION 5 

COMMUNICATION 5 
COMMUNICATION 5 
COMMUNICATION 5 
COMMUNICATION 4 
COMMUNICATION 4 
COMMUNICATION 4 
COMMUNICATION 4 
COMMUNICATION 4 
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COMMUNICATION (Continued) 



Can converse easily 

Eloquent speaker 

Relaxed group speaker 

Is at ease when speaking 

Presents \. -°.tter in a comprehensive 
and interesting manner 

Speaks correctly 

Speaks logically 

Verbal expression is compre^ 
hensive 

Capable in expressing himself 
Difficulty speaking to large groups 



COMMUNICATION 4 
COMMUNICATION 4 
COMMUNICATION 3 
COMMUNICATION 3 
COMMUNICATION 3 

COMMUNICATION 3 
COMMUNICATION 3 
COMMUNICATION 3. 

COMMUNICATION 3 
COMMUNICATION 2 



A ratee's ability to communicate may be expressed by the evaluator with 
one or more descriptive phrases, but this part of the sentence should be 
treated as a single entity and labeled only once with COMMUNICATION. 
For example, M He utilizes a well-rounded vocabulary to very effectively 
express himself." COMMUNICATION 4. 

Instructor would be indexed with COMMUNICATION unless it was the ratee's 
job title or main job duty. In this case it most likely would be labeled 
MANAGEMENT FUNCTIONS or PRODUCTIVITY AND ACHIEVEMENT, depending upon the 
statement. 
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CONDUCT, INTEGRITY, AND PRIDE refers to the way that one acts and to the men- 
tal activities and attitudes that influence behavior, such as integrity 
and pride. Integrity refers to one's moral principles, honesty, and 
sincerity. Pride is the sense of one's own dignity or worth. 

For the assessment of integrity we would look at an individual's use of 
time and expense reports, dealings with others, probity in handling 
assignments, sincerity about his work, and his attitude toward life. 

EXAMPLES : 



His conduct and personal appearance 
are always superb. 

He maintains an exemplary military 
appearance, and has high moral 
standards and sound sense of values. 



CONDUCT, INTEGRITY, AND PRIDE 5 
GROOMING AND ATTIRE 5 

GROOMING AND ATTIRE 4 

CONDUCT, INTEGRITY, AND PRIDE 4 

CONDUCT, INTEGRITY, AND PRIDE 4 



Ratee reflects pride in his position 
as a Chief Petty Officer and pro- 
vides an outstanding example through 
his exceptionally fine personal 
habits and dress under all circum- 
stances . 



CONDUCT, INTEGRITY, AND PRIDE 3 
CONDUCT, INTEGRITY, AND PRIDE 5 
GROOMING AND ATTIRE 5 



His even disposition has proven an 
asset in maintaining an open channel 
of communication between instructor 
and student. 



CONDUCT, INTEGRITY, AND PRIDE 3 
USE OF COMMUNICATION 3 



The ratee has the potential to be- 
come an outstanding chief but has 
problems controlling his drinking 
while on the bench, resulting in 
tardiness at the expiration of 
liberty and his absence during 
working hours. 



POTENTIAL 5 

CONDUCT, INTEGRITY, AND PRIDE 2 

CONDUCT, INTEGRITY, AND PRIDE 2 

CONDUCT, INTEGRITY, AND PRIDE 2 



Excesses in alcohol have led to 
tardiness and a question of his 
dependability . 



CONDUCT, INTEGRITY, AND PRIDE 1 
CONDUCT, INTEGRITY, AND PRIDE 2 
RELIABILITY AND DEPENDABILITY 2 



Additional Indexing Clues : 



He provides an excellent example 
for his men. 



CONDUCT, INTEGRITY, AND PRIDE 5 



Chief XX 1 s conduct is never 
questioned. 



CONDUCT, INTEGRITY, AND PRIDE 5 



Exemplary behavior 



CONDUCT, INTEGRITY, AND PRIDE 4 
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CONDUCT, INTEGRITY, AND PRIDE (Continued) 



Professional behavior 


CONDUCT, INTEGRITY, 
PROFESSIONALISM 3 


AND 


PRIDE 


4 


He is always correct and proper 
in all relationships. 


CONDUCT, 
CONDUCT, 


INTEGRITY, 
INTEGRITY, 


AND 
AND 


PRIDE 
PRIDE 


4 


Quick humor 


CONDUCT, 


INTEGRITY, 


AND 


PRIDE 


4 


Gentlemanly 


CONDUCT, 


INTEGRITY, 


AND 


PRIDE 


3 


At ease with superiors and sub- 
ordinates 


CONDUCT, 


INTEGRITY, 


AND 


PRIDE 


3 


Tactful 


CONDUCT, 


INTEGRITY, 


AND 


PRIDE 


3 


Friendly 


CONDUCT, 


INTEGRITY, 


AND 


PRIDE 


3 


Not afraid to offer criticism 


CONDUCT, 


INTEGRITY, 


AND 


PRIDE 


3 


Bordered on insubordination 


CONDUCT, 


INTEGRITY, 


AND 


PRIDE 


2 



It is important to remember that no matter how positive a trait may be, 
it is impossible to be consistent in weighting these traits; therefore, 
the weight is determined by the adjective which further qualifies the 
type of trait that a ratee possesses. For example, cheerful or cheerful 
personality would be given a weight of 3, while very cheerful or a very 
cheerful personality would be given a weight of 4. 

Also included in this term are personality traits and attitudes. The 
following in their present form would all be indexed as CONDUCT, INTEGRITY 
AND PRIDE 3: optimistic attitude; pleasant attitude; takes pride in him- 
self; pride in his performance; dignified; self-confident; upright; honest 
sincere; does not procrastinate; spends not excessive time "visiting"; 
perfectionist; quiet; unselfish; firm; courage; composed; calm; courtesy; 
disposition; obedience; loyalty to his superiors; friendly. 

Good humor and a good example for contemporaries both would be indexed as 
CONDUCT, INTEGRITY, AND PRIDE 4 because of the word good. 

The word pride often appears in the narrative text in conjunction with 
the way that a ratee feels about the Navy. Under these circumstances > 
use the index term SERVICE MOTIVATION. 
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CONTROLLING is the measurement of performance against established standards, 
correcting deviations, and assuring accomplishment of plans. In simple 
terms, controlling makes certain that what is done is what is intended. 
Forward-looking control prevents deviations from occurring by anticipat- 
ing that they will occur unless action is taken now, su^h as maintenance 
of equipment. 

Any activity which involves feedback is a controlling function in that 
it helps to ensure the successful accomplishment of goals (e.g., checks 
and balances, bookkeeping, accounting systems, traffic flow, and inven- 
tory) . 

EXAMPLES : 

His ability to seek out potential CONTROLLING 5 

problems and correct them before 
failure occurred has been extremely 
beneficial. 



He is consistently capable of CONTROLLING 4 

resolving problem areas before a 
critical situation can develop. 



In his capacity as water chief 
tender he has enforced a strict 
and professional water chemistry 
program. 



CONTROLLING 4 



His analysis of divisional prob- 
lems, both functional and adminis- 
trative, and the execution of 
corrective measures have been very 
conducive to a smooth and highly 
effective division. 



PLANNING 3 
CONTROLLING 3 

PRODUCTIVITY AND ACHIEVEMENT 



Because he never examined the CONTROLLING 1 

fireroom equipment during the 10 
days in port, the unsatisfactory 
condition of the boilers went 
unchecked . 



Additional Indexing Clues : 
Prevents problems 
Corrects difficulties 
Overcomes obstacles 



CONTROLLING 3 
C0NTE.0LLING 3 
CONTROLLING 3 



Maintenance is a controlling function because it helps to assure accom- 
plishment of plans. However, if an individual's job title or main job 
duty is maintenance, then this usually would be indexed as PRODUCTIVITY 
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CONTROLLING (Continued) 

AND ACHIEVEMENT since maintenance is the performance of his total job 
function. 

If the correction, problem solving, or prevention is of a technical nature 
and is performed solely by the rat- e instead of by a group of his subordi- 
nates, it should be i~egarded more as a technical function or r the demon- 
stration of a technical skill rather than considered to be a controlling 
function, and would be labeled TECHNICAL SKILLS, For example, "Ratee's 
knowledge of the P-3 aircraft electronic systems and his ability to expe- 
ditiously correct the most complex electronic problem is outstanding." 
TECHNICAL SKILLS 5; RESPONSIVENESS 3; TECHNICAL SKILLS 5. The modifying 
adverb outstanding belongs to both phrases of this sentence. 

The words assure and ensure should serve as a clue that the phrase which 
follows probably will be indexed as a controlling function. 
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COOPERATION is acting or working together with others or another for a common 
purpose. Includes congenial 7 amiable, or agreeable attitudes which en- 
hance the act of working together. 

EXAMPLES: 



He always cooperates fully with his 
seniors and accomplishes his duties 
in an outstanding manner. 

He is always willing to help others 
in any way he can. 

Ratee exemplifies the perfect 
officer-chief relationship . 

Ratee is very cooperative with his 
seniors and his contemporaries, 
always putting the interests of 
the Navy first. 

He is cooperative and well liked. 

He is a pleasure to work with. 

He will nevertheless listen to the 
ideas, beliefs, and suggestions of 
others. 

Additional Indexing Clues : 

Never hesitates to fully cooperate 

Works well with superiors and 
subordinates 

Gets along well with others 

Continuous cooperation in all aspects 

Complete cooperation 

Congenial 

Amiable 

Cooperative 

He will compromise. 

Cooperative with superiors and 
subordinates 

Assists others 

Uncooperative 



COOPERATION 5 

PRODUCTIVITY AND ACHIEVEMENT 5 

COOPERATION 5 

COOPERATION 5 

COOPERATION 4 
SERVICE MOTIVATION 5 



COOPERATION 3 
REPUTE 4 

COOPERATION 3 

COOPERATION 3 



COOPERATION 5 
COOPERATION 4 

COOPERATION 4 
COOPERATION 4 
COOPERATION 4 
COOPERATION 3 
COOPERATION 3 
COOPERATION 3 
COOPERATION 3 
COOPERATION 3 

COOPERATION 3 
COOPERATION 2 



Usually a ratee' s interpersonal relationship with his subordinates would 
be indexed as LEADERSHIP AND DIRECTING. However, when a statement has 
to do with a ratee' s cooperative attitude in working with others, whether 
superiors or subordinates, then it should be labeled COOPERATION. 
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DRIVE is the desire or personal motivation to achieve a purpose. Includes the 
desire for self-improvement or the desire to manage, achieve a purpose, 
and improve output through the teamed effort of subordinates. Includes 
conscientious , energetic, enthusiasm , forceful, and interest. 

EXAMPLES: 



He is an exceptionally dedicated 
and hard working Chief, readily 
accepting and expeditiously 
solving problems. 



DRIVE 5 
DRIVE 5 

RELIABILITY AND DEPENDABILITY 
CONTROLLING 4 



Always works to his fullest 
potential • 



DRIVE 5 



This highly energetic and enthusi- 
astic young petty officer has per- 
formed all aspects of his duties 
in an outstanding fashion. 



DRIVE 4 
DRIVE 4 

PRODUCTIVITY AND ACHIEVEMENT 5 



Ratee performs with vigor , 



DRIVE 3 



He is conscientious and is always 
extremely dependable. 



DRIVE 3 

RELIABILITY AND DEPENDABILITY 5 



Although the tasks assigned to 
the personnel under his supervi- 
sion are always completed effi- 
ciently and in an excellent to 
outstanding manner, his superiors 
feel that he is capable of getting 
more out of his men and of putting 
more of himself into the job. 



PRODUCTIVITY AND ACHIEVEMENT 5 
LEADERSHIP AND DIRECTING 2 
DRIVE 2 



Ratee is not as aggressive as he 
could be which subsequently de- 
tracts from him reaching his full 
potential. 



DRIVE 2 
POTENTIAL 2 



Additional Indexing Clues : 



He continually strives for per- 
fection. 



DRIVE 5 



Always driving to do the best 
job possible 



DRIVE 5 



Completely dedicated 

Attempts to achieve perfection 



DRIVE 4 
DRIVE 4 



Volunteers for jobs 



DRIVE 3 
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DRIVE (Continued) 

The following words and ph/ases would be indexed as DRIVE 4: unflagging 
effort; works very hard at his job; spends many extra hours at his job; 
dedicates or devotes long hours; tireless; chief is tireless in his ef- 
forts to improve his equipment, personnel, and records; always willing 
to work long hours whenever necessary; continuously strives; taking courses 
in off-duty hours. 

The following words and phrases would be indexed as DRIVE 3: intent on 
doing job well; determined; works hard; applied himself; strives for im- 
provement; drive; working long hours; aggressive pursuit of duties; 
can-do spirit or attitude; diligent; dynamic; ambitious; zeal; endeavor. 

When a statement about drive is part of how a ratee is performing a 
specific function or skill, then the statement is used as a qualifier 
for that function or skill. For example, "Ratee works very hard at moti- 
vating his men to improve their performance/ 1 would be indexed as LEADER- 
SHIP AND DIRECTING 4, and DRIVE would not be used. However, a statement 
such as, n Ratee is constantly seeking to improve the productivity of the 
work center," would be indexed as DRIVE 4 since there is no indication 
that this objective has been achieved and there is no mention of any 
specific function or skill. 
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ENDURANCE is the ability to complete tasks under conditions of fatigue, dis- 
tress, stress, and/or pain. 



EXAMPLES : 

His endurance is outstanding. 

During the previous deployment he 
demonstrated his ability to func- 
tion smoothly and effectively under 
adverse and demanding conditions 
for extended periods. 

His ability to maintain an inner 
calm and to function efficiently 
during periods of great confusion 
and stress suit him ideally to his 
present assignment. 

He uncomplainingly responds when 
called upon to meet unscheduled 
commitments, frequently under 
adverse conditions. 

Additional Indexing Clues : 

Performs well under stress 

His persistence (tenacity) in 
the face of adversity 

Fortitude 

Indefatigable 

Tested and proven under hostile 
fire 



ENDURANCE 5 
ENDURANCE 4 



CONDUCT, INTEGRITY, AND PRIDE 3 
ENDURANCE 4 



RESPONSIVENESS 4 
ENDURANCE 3 



ENDURANCE 4 
ENDURANCE 3 

ENDURANCE 3 
ENDURANCE 3 
ENDURANCE 3 



Perseverance 



ENDURANCE 3 
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FLEXIBILITY is the quality of being adj 
of modif ication. 

EXAMPLES : 

His keen mind is alert to all 
possible circumstances, and he 
succeeds brilliantly in adjusting 
to new environments. 

Overall, he is a highly adaptable 
individual who exhibits unlimited 
potential and continuing high 
value to the U.S. Navy. 

He is a mature, stable Chief 
Petty Officer, intelligent, 
adaptable, and reliable. 



Additional Indexing Clues : 
Adjusts quickly 

Able to handle varied jobs and 
hence is a very valuable Chief 

Flexible 

Open minded 

Open mind to criticism 



stable or adaptable to change; capable 



INTELLECTUAL FUNCTIONING 4 
FLEXIBILITY 5 



FLEXIBILITY 4 

POTENTIAL 5 

ASSET TO THE NAVY 4 



RELIABILITY AND DEPENDABILITY 3 
RELIABILITY AND DEPENDABILITY 3 
INTELLECTUAL FUNCTIONING 3 
FLEXIBILITY 3 

RELIABILITY AND DEPENDABILITY 3 



FLEXIBILITY 4 

FLEXIBILITY 3 
ASSET TO THE NAVY 4 

FLEXIBILITY 3 

FLEXIBILITY 3 

FLEXIBILITY 3 
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GROOMING AND ATTIRE is the way in which a 
appearance and clothing. 

EXAMPLES : 

Neat and polished appearance is in 
keeping with the highest Navy stan- 
dards and serves as a criteria of 
excellence among the men with whom 
he comes in contact. 

Ratee's personal appearance is 
always correct and proper. 

He is always neat in appearance 
and his conduct is exemplary. 

He is tidy, intelligent, and ob- 
tains the best results from his 
men. 

His tendency towards being over- 
weight greatly detracts from his 
overall appearance. 

Additional Indexing Clues : 

His impeccable appearance leaves 
nothing to be desired. 

Wearing of uniform excellent 

Always impeccable 

His appearance is immaculate 
at all times. 

His appearance and dress is always 
correct , smart, and impressive . 

His uniform is consistently 
immaculate . 

His appearance is good and sets 
an example for all. 

His dress is impressive, and is 
worn with care. 

Uniform (or dress) immaculate 

Grooming impeccable 

He takes pride in his appear- 
ance. 

Attention to his appearance 
Appearance military 
Shined shoes 



person cares for his physical 



GROOMING AND ATTIRE 5 



GROOMING AND ATTIRE 4 

GROOMING AND ATTIRE 4 

CONDUCT, INTEGRITY, AND PRIDE 4 

GROOMING AND ATTIRE 3 
INTELLECTUAL FUNCTIONING 3 
LEADERSHIP AND DIRECTING 5 

GROOMING AND ATTIRE 2 



GROOMING AND ATTIRE 5 

GROOMING AND ATTIRE 5 
GROOMING AND ATTIRE 4 
GROOMING AND ATTIRE 4 

GROOMING AND ATTIRE 4 

GROOMING AND ATTIRE 4 

GROOMING AND ATTIRE 4 

GROOMING AND ATTIRE 4 

GROOMING AND ATTIRE 4 
GROOMING AND ATTIRE 4 
GROOMING AND ATTIRE 3 

GROOMING AND ATTIRE 3 
GROOMING AND ATTIRE 3 
GROOMING AND ATTIRE 3 
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INITIATIVE is the ability to recognize and originate necessary or appropriate 
tasks on one's own without being asked. 

EXAMPLES ; 



His personal appearance, cooperative 
attitude, and initiative leave vir- 
tually nothing to be desired. 

He is extremely reliable and never 
fails to take the initiative in 
difficult situations. 

He displays an outstanding example 
in initiative. 

Ratee displays considerable initia- 
tive in accomplishing each task 
assigned, 

Ratee possesses a keen mind, is 
capable of original thinking, and 
expresses his thoughts well and 
decisively when communicating 
with others. 

Ratee is proficient in anticipat- 
ing situations in his area of 
Lesponsibility and initiates 
action to cope with it. 

Ratee undertakes his duties will- 
ingly but lacks the initiative 
that is required of the very best 
in his rate. 

However, at times he lacks the 
initiative and drive that is neces- 
sary to produce these results. 

Additional Indexing Clues : 

Provided Division officer with 
many ideas 

Volunteering his own views 

Making suggestions (recommendations) 

An original thinker 

Institutes 



GROOMING AND ATTIRE 5 
COOPERATION 5 
INITIATIVE 5 

RELIABILITY AND DEPENDABILITY 5 
INITIATIVE 5 

INITIATIVE 5 

TIATIVE 4 



INTELLECTUAL FUNCTIONING 3 
INITIATIVE 3 
COMMUNICATION 4 



CONTROLLING 4 
INITIATIVE 3 



RELIABILITY AND DEPENDABILITY 4 
INITIATIVE 2 



INITIATIVE 2 
DRIVE 2 



INITIATIVE 4 

INITIATIVE 3 
INITIATIVE 3 
INITIATIVE 3 
INITIATIVE 3 



"He has shown initiative by repairing the probe refueling hose and fixing 
the main feed pumps in the forward fireroom." INITIATIVE 3; PRODUCTIVITY 
AND ACHIEVEMENT 3- Here the index term PRODUCTIVITY AND ACHIEVEMENT is 
used rather than TECHNICAL SKILLS because the indexer cannot b£ certain 
whether the ratee had the skill to do this himself or whether, he actually 
managed or directed others to accomplish this task. CONTROLLING is not 
used here because the ratee* s main job duty was maintenance. 
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INTELLECTUAL FUNCTIONING is the ability to learn or understand from experience, 
and the ability to analyze, reason, and perceive relationships and differ- 
ences. 

A measurement of intellectual functioning would be a scholastic record. 



EXAMPLES: 



He completed all courses with very 
high grades, usually leading his 
class. 

Ratee is extremely intelligent .« 

He learns quickly and applies his 
training and experience effectively. 

He is a forceful and intelligent 
Career Petty Officer who has such 
command of the basic professional 
techniques that he can direct his 
actions to job perfection and the 
well-being of his subordinates. 

He is tidy 5 intelligent, and 
obtains the best results from 
his men. 



INTELLECTUAL FUNCTIONING 5 



INTELLECTUAL FUNCTIONING 5 

INTELLECTUAL FUNCTIONING 4 
PRODUCTIVITY AND ACHIEVEMENT 4 

DRIVE 3 

INTELLECTUAL FUNCTIONING 3 
PROFESSIONALISM 4 
TECHNICAL SKILLS 4 
PRODUCTIVITY AND ACHIEVEMENT 5 
LEADERSHIP AND DIRECTING 4 

GROOMING AND ATTIRE 3 
INTELLECTUAL FUNCTIONING 3 
LEADERSHIP AND DIRECTING 5 



Additional Indexing Clues : 



Learned quickly 


INTELLECTUAL 


FUNCTIONING 


4 


Coherent mental organization 


INTELLECTUAL 


FUNCTIONING 


3 


Coherence of his thoughts 


INTELLECTUAL 


FUNCTIONING 


3 


Intelligence 


INTELLECTUAL 


FUNCTIONING 


3 


Keen minded 


INTELLECTUAL 


FUNCTIONING 


3 


Common sense 


INTELLECTUAL 


FUNCTIONING 


3 


Judgment 


INTELLECTUAL 


FUNCTIONING 


3 


Insight 


INTELLECTUAL 


FUNCTIONING 


3 



Alert would be indexed as INTELLECTUAL FUNCTIONING 3. However, if the 
ratee is alert to a command or an order, it would be indexed as RESPON- 
SIVENESS 3. 

Analytical mind would be indexed as INTELLECTUAL FUNCTIONING 3. Some- 
times intellectual functioning, planning, and controlling become confused. 
It is important here to keep in mind whether the rater was referring to 
the function of planning or controlling or whether he was making a state- 
ment about the ratee' s intellectual functioning, i.e., his general analyti- 
cal ability, his ability to proceed logically, or his problem-solving abili- 
ty. For example, "His imagination allows him to find new and different 
solutions to problems which others do not seem to be able to solve," would 
be indexed as RESOURCEFULNESS 3 and INTELLECTUAL FUNCTIONING 4. 
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LEADERSHIP AND DIRECTING represent motivating, guiding, and supervising of 

subordinates to accomplish a job. and work towards improved performance. 
Includes encouraging subordinates in cooperative endeavors and also in 
self -development through counseling. A measure would be vaorale. 

LEADERSHIP AND DIRECTING also would be creating an atmosphere that makes 
teamwork possible, such as improving working conditions. 

EXAMPLES : 



His excellent leadership qualities 
were especially manifest when he 
was assigned to lead the mainte- 
nance efforts on CQ. detachments in 
CONSTELLATION and INDEPENDENCE. 



LEADERSHIP AND DIRECTING 5 



Ratee handles his men in an effec- 
tive manner and always gets good 
results from his subordinates. 



LEADERSHIP AND DIRECTING 4 
LEADERSHIP AND DIRECTING 4 



Ratee 1 s knowledge of the S-2E 
Electronic System is outstanding, 
and he is able to utilize this 
exceptional knowledge through 
skillful management of shop person- 
nel and an innate ability to pass 
along what he knows to others. 



TECHNICAL SKILLS 5 
TECHNICAL SKILLS 5 
LEADERSHIP AND DIRECTING 4 
COMMUNICATION 4 



He spent a great deal of extra time 
with his men and turned out well 
trained, well motivated men for the 
fleet. 



LEADERSHIP AND DIRECTING 4 
STAFFING 4 

LEADERSHIP AND DIRECTING 4 



Ratee has the ability to solve prob- 
lems and motivate people. 

His loyalty was shown in the con- 
scientious manner in which he attend- 
ed to the problems of his men. 

He appears to be indifferent to 
the personnel administration of 
his men especially in regard to 
special requests and advancement. 

Although the tasks assigned to the 
personnel under his supervision are 
always completed efficiently and in 
an excellent to outstanding manner, 
his superiors feel that he is capa- 
ble of getting more out of his men 
and of putting more of himself into 
the job. 



INTELLECTUAL FUNCTIONING 3 
LEADERSHIP AND DIRECTING 3 

CONDUCT, INTEGRITY, AND PRIDE 3 
LEADERSHIP AND DIRECTING 3 



LEADERSHIP AND DIRECTING 2 



PRODUCTIVITY AND ACHIEVEMENT 5 
LEADERSHIP AND DIRECTING 2 
DRIVE 2 
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LEADERSHIP AND DIRECTING (Continued) 



Additional Indexing Clues: 



Gets most out of them 


LEADERSHIP 


AND 


DIRECTING 


5 


Best results from men 


LEADERSHIP 


AND 


DIRECTING 


5 


Consistent in direction of 

■n oTQnnn ol 

pcL oUllllfc;X 


LEADERSHIP 


AND 


DIRECTING 


4 


11 OllU p CD U L XI- 


T FADFRSHTP 

J_i L-lCxXJ ill 1\ lJ ILd-L 


AND 


DTR "RPTTMn 

JJ X 1\ Hi V> _L XINvJ 


A 


Hi c*h tppatH "FnT* mpn 

llXg 11 L Cg CL i- \J. J_ 1_ IL1C LI 


LEADERSHIP 


AND 


DTRFCTTNG 

I^XIXDVjI X 1 1 VJ 


4 


Aggressive leadership 


LEADERSHIP 


AND 


DIRECTING 


4 


Molded crew into competent ano 
effective team 


LEADERSHIP 


AND 


DIRECTING 

j_y X. i\ XJ V> J. X. L 1 \-J 


4 


Directs men well 


LEADERSHIP 


AND 


DIRECTING 


4 


High expectations from men 


LEADERSHIP 


AND 


DIRECTING 


4 


High regard for team concept 


LEADERSHIP 


AND 


DIRECTING 


4 


Promotes harmony and accord 


LEADERSHIP 


AND 


DIRECTING 


4 


Mpti phppT"fn1 

ll^Ll LUCC 1_ LUX 


LEADERSHIP 


AND 


DIRECTING 


3 


LcU Lilt. DI1U p 


LEADERSHIP 

Xj LJJTxlJ iluL\ u 11X1 


AND 


DTRFCTTNG 




DlltUULdgcb d LIU. gUlUco DUUULUlllaUco 


T FADFRSHTP 

J_i Hi t\XJ J_j 1\ lJ 11X1 


anh 


JJ XIvIIiVj L J_1N\J 




1 It, i.pt.U Ulcll CtvJ. VCtLlL-t: Xll L a LC 


T FADFRSHTP 

J_i Hi t\XJ Hi 1\ lJ 1 1 X 1 


AND 


DTRFPTTNO 

JJ X 1\ Hi Vj X X IN VJ 




Leads by setting the example 


LEADERSHIP 


AND 


DIRECTING 


3 


Leadership ability 


LEADERSHIP 


AND 


DIRECTING 


3 


Men trust him/loyal to him 


LEADERSHIP 


AND 


DIRECTING 


3 


Finds time to direct and 
counsel young men 


LEADERSHIP 


AND 


DIRECTING 


3 


Skilled at managing his men 


LEADERSHIP 


AND 


DIRECTING 


3 


Tact in handling subordinates 


LEADERSHIP 


AND 


DIRECTING 


3 


His men are hard working 


LEADERSHIP 


AND 


DIRECTING 


3 


No disciplinary problems 


LEADERSHIP 


AND 


DIRECTING 


3 


Mindful of his position as a 
leader 


LEADERSHIP 


AND 


DIRECTING 


3 


Lack of leadership 


LEADERSHIP 


AND 


DIRECTING 


2 


Needs more forceful approach 


LEADERSHIP 


AND 


DIRECTING 


2 



to leadership 

Many experts in the field of management feel that the connection between 
performance and possession of traits is doubtful. But practically every 
study has found successful managers to be strong leaders. Leadership is 
not only an ability or trait but it is also a very important function of 
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LEADERSHIP AND DIRECTING (Continued) 

management. To direct his subordinates, a manager must motivate, com- 
municate, and lead. Directing is a function that includes all those 
activities which are designed to encourage subordinates to work effec- 
tively and efficiently. A manager must be concerned for human feelings 
and morale. For example, "He is tactful with his subordinates . " It can 
be seen here that interpersonal relations betvc a ratee and his sub- 
ordinates is important to the leadership and d ting function. The 
rapport that a ratee has with his divisional pei^onnel would be indexed 
with LEADERSHIP AND DIRECTING. However, a ratee's interpersonal rela- 
tionship with his peers or superiors probably would be indexed by one of 
the following terms; CONDUCT, INTEGRITY, AND PRIDE; COOPERATION; REPRE- 
SENTATION; or RESPONSIVENESS. There is one exception to this rule. When 
the statement has to do with a ratee T s cooperative attitude in working 
with others, superiors or subordinates, use the index term COOPERATION. 
The rapport that a ratee has with other organizational units enhances his 
division 1 s work and, therefore, the index term REPRESENTATION would be 
used . 

A manager must act his part and be conscious of the impact of his be- 
havior on his men. For exampJ ; "He leads by example." 

A manager also must look after * individual needs of his subordinates 
and provide a challenge for theiu. He must be responsive to their needs 
and to their ideas and suggestions. 

The words supervising or directing can apply to tasks, men, or both. The 
indexer has to judge what the rater means. It usually can be assumed that 
supervising refers to the overall managerial functions or tasks (MANAGE- 
MENT FUNCTIONS), unless stated or inferred otherwise. 
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MANAGEMENT FUNCTIONS are those job duties which are characteristic of all 

managers. Though operations may differ from one organization to another, 
the functions of the manager are common to all. 



EXAMPLES : 

He is an excellent manager and 
organizer who is willing to accept 
any assignment no matter how 
difficult . 

Chief XX has made a prime contribu- 
tion to the ship through his excel- 
lent supervision of the Fuel Oil 
and Water Testing Laboratory and 
the Oil Kings. 

Ratee's superior leadership capa- 
bilities and overall knowledge of 
management greatly contributed to 
this division receiving a grade of 
4.0 during the annual administra- 
tive inspection. 

He is well versed in the 3-M System 
and always exhibits sound manage- 
ment practices. 

His administrative knowledge and 
ability to supervise and coordinate 
the efforts of other instructors 
enabled this command to develop 
all the material required for 
realistic support of the E2B 
Aircraft. 



MANAGEMENT FUNCTIONS 5 
ORGANIZATION 5 

RELIABILITY AND DEPENDABILITY 5 



PRODUCTIVITY AND ACHIEVEMENT 4 
MANAGEMENT FUNCTIONS 5 



LEADERSHIP AND DIRECTING 5 
MANAGEMENT FUNCTIONS 4 
PRODUCTIVITY AND ACHIEVEMENT 4 



TECHNICAL SKILLS 4 
MANAGEMENT FUNCTIONS 4 



MANAGEMENT FUNCTIONS 3 
LEADERSHIP AND DIRECTING 3 
ORGANIZATION 3 

PRODUCTIVITY AND ACHIEVEMENT 4 



BTC has an excellent working and 
practical knowledge of the PMS 
System but has a tendency to be 
lax in the administrative phase 
of the system. 

Additional Indexing Clues : 

Proficient Petty Officer 



TECHNICAL SKILLS 5 
MANAGEMENT FUNCTIONS 



MANAGEMENT FUNCTIONS 4 



The words supervising or directing can apply to tasks or, men. The indexer 
has to judge what the rater means. It usually can be assumed that super- 
vising refers to the overall managerial functions or tasks, unless stated 
or inferred otherwise. If the statement refers to the supervising or 
directing of men only, then use LEADERSHIP AND DIRECTING. 
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MANAGEMENT FUNCTIONS (Continued) 

E-7/8/9 Evaluation Reports describe the on-job performance of junior 
level managers. Managers at a junior level have technical as well as 
• managerial functions. This should be remembered when indexing a state- 
ment that includes the ratee T s job title* For example, if the statement 
claims that the ratee is an "excellent Radioman or Boilerman," we would 
assume that the rater was referring to his technical skills and label it 
as such. But on the other hand, if a statement claims that the ratee 
is an "excellent Chief Petty Officer or Supervisor , !! it would be assumed 
that the rater is referring more to the ratee' s managerial functions and 
would be labeled accord in gly. Also remember that if Instructor is a 
ratee's job title, a statement regarding his instructing would be labeled 
as MANAGEMENT FUNCTIONS, whereas if the reference is to how a ratee in- 
structs during the course of his other duties, use the label COMMUNICA- 
TION, 

In evaluating a ratee, the indexer should be interested in how the ratee 
performs his managerial functions. These functions are the means by 
which the manager proceeds to accomplish his job. When a statement re- 
fers to how he performs these overall tasks, use MANAGEMENT FUNCTIONS, 
However, when a statement refers to the results accomplished from per- 
forming these tasks, use PRODUCTIVITY AND ACHIEVEMENT, 
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ORGANIZATION is the establishment of an intentional structure of roles through 
the determination and enumeration of activities required to achieve enter- 
prise goals such as grouping activities and roles, delegating authority, 
and coordinating authority relationships. 

EXAMPLES : 



He does an excellent job of 
planning, organizing, and carry- 
ing out his job* 

He has developed an extremely 
tightly knit division which has an 
uncommon amount of pride in its work. 



PLANNING 5 
ORGANIZATION 5 

PRODUCTIVITY AND ACHIEVEMENT 5 

ORGANIZATION 5 

LEADERSHIP AND DIRECTING 5 



He is a highly motivated and 
aggressive individual with a good 
sense or organization and adminis- 
trative ability. 



DRIVE 4 
DRIVE 4 
ORGANIZATION 4 
MANAGEMENT FUNCTIONS 



Ratce spends a very limited time in 
the shop but has exerted a spirit of 
independence in his First Class, the 
result being a well organized and 
efficient shop. 



DRIVE 2 

LEADERSHIP AND DIRECTING 4 
ORGANIZATION 4 

PRODUCTIVITY AND ACHIEVEMENT 



His ability to assign workload in a 
smooth fashion is noteworthy. 



ORGANIZATION 4 



He is a professional administrator 
and understands the principles of 
delegation. 



PROFESSIONALISM. 3 
MANAGEMENT FUNCTIONS 4 
ORGANIZATION 3 



The ratee's ability to work with 
others, his capacity for organi- 
zation and stimulating enthusiasm 
makes him a valuable asset to any 
unit . 



COOPERATION 3 
ORGANIZATION 3 
DRIVE 4 

ASSET TO THE NAVY 4 



He reorganized the work center. 

His initiative at times lags, and 
he tends to undertake too many 
tasks alone rather than delegating 
them to subordinates. 



ORGANIZATION 3 

INITIATIVE 2 
ORGANIZATION 2 



Additional Indexing Clues : 
Excellent organizer 



OP 3ANIZATI0N 5 



Excellent job setting up the 
operation 



ORGANIZATION 5 
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ORGANIZATION (Continued) 



Set up a file on each item 



ORGANIZATION 3 



Ability to coordinate 



ORGANIZATION 3 



Coordinates work centers 



ORGANIZATION 3 
0RGANIZATI0N)3 



Coordination of work duties 




If a ratee sets up a liaison with another organizational unit or division 
within the Navy or an outside organization, the index term ORGANIZATION 
would be used. However, if a ratee uses the liaison or organizational 
structure that is already set up to enhance his division's working rela- 
tionship with other organizational units, the index term REPRESENTATION 
would be used. If a liaison is used as a channel of communication, then 
the index terra USE OF COMMUNICATION would be used. 
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PLANNING is a decision-making process involving the selection among alterna- 
tives of objectives, policies, and programs and the procedures for achiev 
ing them. 

Types of plans include objectives, policies, methods or procedures, rules 
programs, budgets, strategies, schedules, traffic flow patterns, and in- 
ventory. 

EXAMPLES : 



He does an excellent job of plan- 
ning, organizing, and carrying 
out his job. 

He is methodical, deliberate, and 
able to develop effective and 
efficient procedural methods of 
accomplishing the division work- 
load . 



PLANNING 5 
ORGANIZATION 5 

PRODUCTIVITY AND ACHIEVEMENT 5 

SKILLS AND ABILITIES 3 
SKILLS AND ABILITIES 3 
PLANNING 4 



Ratee's resourcefulness in setting 
up a procedure to instruct 150 
recruits daily in this very diffi- 
cult operation was largely re- 
sponsible for the efficient 
operation of the matches. 

He had the ability and initiative 
to plan and assign work to person- 
nel, under his direction to ensure 
that the end results are of the 
highest quality. 

His analysis of divisional prob- 
lems, both functional and adminis- 
trative, and the execution of 
corrective measures have been 
very conducive to a smooth and 
highly effective division. 

Additional Indexing Clues: 



RESOURCEFULNESS 3 
PLANNING 4 

PRODUCTIVITY AND ACHIEVEMENT 



INITIATIVE 3 
PLANNING 3 
ORGANIZATION 3 
CONTROLLING 5 



PLANNING 3 
CONTROLLING 3 

PRODUCTIVITY AND ACHIEVEMENT 



Found new ways to improve the 
security of the Communications 
building 

Good decision maker 
Improves plans 



PLANNING 4 

PLANNING 4 
PLANNING 4 



Has alternate plans 



PLANNING 3 
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POTENTIAL refers to the capacity and/or recommendation for an individual to 
assume a higher rank or added responsibilities, 

EXAMPLES: 



Chief XX has demonstrated the 
potential to be an outstanding 
leader. 



POTENTIAL 5 



He would be most valuable in an 
instructor billet assignment. 



POTENTIAL 5 



There is nothing that can arise in 
his present position or next assign- 
ment that ratee cannot cope with. 



POTENTIAL 5 



He is highly recommended for 
advancement. 



POTENTIAL 4 



Chief XX is highly recommended for 
advancement and retention in the 
Naval Reserve, 



POTENTIAL 4 



He is qualified to assume the 
greater responsibility of the 
next higher pay grade. 



SKILLS AND ABILITIES 3 
POTENTIAL 3 



Ratee is fully qualified for 
advancement in rate. 



SKILLS AND ABILITIES 4 
POTENTIAL 3 



Ratee is very knowledgeable in the 
supply system and is recommended 
for E-8. 



TECHNICAL SKILLS 4 
POTENTIAL 3 



With more time and conscientious 
effort, he should realize a 
greater potential. 



POTENTIAL 2 



Additional Indexing Clues : 

Outstanding potential for added 
responsibility 

Highly recojnmended for advancement 

Strongly recommended for promocion 

Eminently well qualified for 
advancement \ 

Expected to improve 

Capable of assuming more responsi- 
bility 



POTENTIAL 5 

POTENTIAL 4 
POTENTIAL 4 

SKILLS AND ABILITIES 4 
POTENTIAL 3 

POTENTIAL 3 

POTENTIAL 3 
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PRODUCTIVITY AND ACHIEVEMENT refers to the successful accomplishment of a 

desired result in terms of an individual's performance of his job duties 
(managerial functions). Use this descriptor term unless a specific 
function is mentioned in the narrative content of the Evaluation Report. 

EXAMPLES : 

Every assignment is performed with PRODUCTIVITY AND ACHIEVEMENT 5 

unusual accuracy and effectiveness. 

Ratee's performance is outstanding PRODUCTIVITY AND ACHIEVEMENT 5 

in all aspects. 

Performance in both his primary and PRODUCTIVITY AND ACHIEVEMENT 5 

secondary billets has been outstand- 
ing in all aspects as marked in 
block 19. 



Under his supervision the shop has 
met and surpassed the accepted 
norms of productivity. 

Production over the past six months 
has been about average for a two- 
man station. 

His technical competence and re- 
sourcefulness contributes to ships 
in tending to maintain operational 
readiness . 

His performance has been commen- 
surate with his rate. 

While he was acting as ship's Oil 
King, the ship witnessed three 
oil spills while refueling. 

His extremely poor performance 
of his duties led directly to the 
ship's boilers not being in a 
state of operational readiness. 

Additional Indexing Clues : 

Never allowed shop to wane 

Outstanding performance 

Handled duties in an outstanding 
manner 

Impressive discharge of duty 



PRODUCTIVITY AND ACHIEVEMENT 4 

PRODUCTIVITY AND ACHIEVEMENT 3 

TECHNICAL SKILLS 3 
RESOURCEFULNESS 3 
PRODUCTIVITY AND ACHIEVEMENT 3 

PRODUCTIVITY AND ACHIEVEMENT 3 

PRODUCTIVITY AND ACHIEVEMENT 2 

PRODUCTIVITY AND ACHIEVEMENT 1 



PRODUCTIVITY AND ACHIEVEMENT 5 
PRODUCTIVITY AND ACHIEVEMENT 5 
PRODUCTIVITY AND ACHIEVEMENT 5 

PRODUCTIVITY AND ACHIEVEMENT 4 
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PRODUCTIVITY AND ACHIEVEMENT (Continued) 



Performed remarkably well 

Success in a difficult task 

Assignments are completed or 
completes assigned tasks 

Has seen these tasks through to 
their successful completion 

Effort brings a success 



PRODUCTIVITY AND ACHIEVEMENT 4 

PRODUCTIVITY AND ACHIEVEMENT 3 

PRODUCTIVITY AND ACHIEVEMENT 3 

PRODUCTIVITY AND ACHIEVEMENT' 3 

PRODUCTIVITY AND ACHIEVEMENT 3 



Statements of improvement should be indexed as PRODUCTIVITY AND ACHIEVE- 
MENT. For example: 



Made significant improvements 
Made improvements 
Enabled him to improve 
Yielded benefits 
Room for improvement 



PRODUCTIVITY AND ACHIEVEMENT 4 
PRODUCTIVITY AND ACHIEVEMENT 3 
PRODUCTIVITY AND ACHIEVEMENT 3 
PRODUCTIVITY AND ACHIEVEMENT 3 
PRODUCTIVITY AND ACHIEVEMENT 2 
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It is important to make certain that the achievement was not a personal 
technical accomplishment in which case TECHNICAL SKILLS would be used. 

If a statement refers to the successful accomplishment of a maintenance 
effort and it is not the controlling function of ensuring the accomplish- 
ment of a plan, but rather the performance of a ratee's total job func- 
tion, use PRODUCTIVITY AND ACHIEVEMENT. This can be determined by look- 
ing at the ratee's job title or main job duty as stated in Section 4A. _ 

The following modifiers can serve as clues to labeling a statement or a 
phrase as PRODUCTIVITY AND ACHIEVEMENT: contributed greatly to (4), 
contributed immensely to (4) , significant increase in (4) , contributed 
materially to (4), directly responsible for (3), directly contributed 
to (3), has been instrumental in (3) v , directly instrumental in (3). 

If a statement of personal achievement (e.g., advancement in rate) is 
made in a matter-of-fact manner, do not index it. However, if it is 
stated as an accomplishment, use PRODUCTIVITY AND ACHIEVEMENT. For 
example, "Ratee T s performance in the past was responsible for his selec- 
tion from every first class and chief petty officer in this command as 
Command Career Counselor." PRODUCTIVITY AND ACHIEVEMENT 5. 

An ability in performing a task would be indexed as PRODUCTIVITY AND 
ACHIEVEMENT because it tells how the ratee is performing and, therefore, 
is a type of measure of his performance. However, an ability or skill to 
perform in a certain way suggests a future performance and would be index- 
ed as SKILLS AND ABILITIES. For example, ,f His thoroughness in performing 
his duty is outstanding," PRODUCTIVITY AND ACHIEVEMENT 5; "Outstanding in 
his work," PRODUCTIVITY AND ACHIEVEMENT 5; "Chief XX is a performer," PRO- 
DUCTIVITY AND ACHIEVEMENT 3; "He performs in a capable manner," PRODUCTIVI- 
TY AND ACHIEVEMENT 3; "He has the ability to perform in an outstanding man- 
ner," SKILLS AND ABILITIES 5. 
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PROFESSIONALISM is the quality of being worthy of the high standards of the 

managerial profession and having much experience and great skill in this 
role. Use only when the word professional or professionalism is applied, 
to the ratee and/or his performance and only when it is mentioned specifi- 
cally in the narrative content of the Evaluation Report. 

EXAMPLES : 



Ratee has consistently demonstrated 
his outstanding professional quali- 
fications. 

These accomplishments were achieved 
through high professionalism, en- 
thusiasm, and superior decision- 
making ability. 

As an Airborne Communications 
Supervisor, ratee r s professional 
competence has contributed to 
flight operations, excellent 
mission performance, and praise 
for his division. 

His performance reflects compe- 
tence and professionalism. 

Chief XX was relieved of his duties 
as the ship's Oil King after serving 
in the capacity for approximately 
two months. He was removed from 
this billet because of his lack of 
professional knowledge and technical 
know-how in the art of refueling. 



PROFESSIONALISM 5 



PROFESSIONALISM 4 
DRIVE 4 
PLANNING 5 



PROFESSIONALISM 3 
PRODUCTIVITY AND ACHIEVEMENT 5 



PRODUCTIVITY AND ACHIEVEMENT 3 
SKILLS AND ABILITIES 3 
PROFESSIONALISM 3 

AWARDS AND PUNISHMENT 1 
PROFESSIONALISM 2 
TECHNICAL SKILLS 2 



Additional Indexing Clues : 
Professional attitude 

Professional behavior 

Professional administrator 

Professional skill 



PROFESSIONALISM 3 

CONDUCT, INTEGRITY, AND PRIDE 4 

PROFESSIONALISM 3 

CONDUCT, INTEGRITY, AND PRIDE 4 

PROFESSIONALISM 3 
MANAGEMENT FUNCTIONS 4 

PROFESSIONALISM 3 



When the word professional modifies a specific skill, ability or function, 
it automatically gives a weight of 4 to that skill, ability or function. 



When the word professional modifies skill or ability, use only the index 
term PROFESSIONALISM since this term is more specific than SKILLS AND 
ABILITIES and falls under SKILLS AND ABILITIES in the hierarchy of index 
terms. 
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RELIABILITY AND DEPENDABILITY is the combined quality of being counted upon to 
do what is expected or required without direct supervision or monitoring. 
Connotes levelheadedness or steadiness; includes trustworthy, responsible , 
constancy , stability , and mature. 



EXAMPLES: 



He is completely reliable and has 
never showed case by action or 
word to the contrary. 



RELIABILITY AND DEPENDABILITY 5 



He is conscientious and is always 
extremely dependable. 

Fully realizes- his responsibili- 
ties and at all times consciously 
acts to fulfill them. 



DRIVE 3 

RELIABILITY AND DEPENDABILITY 5 

RELIABILITY AND DEPENDABILITY 4 
RELIABILITY AND DEPENDABILITY 4 



Ratee is very reliable, 

Ratee is punctual and can be de- 
pended upon to perform well re- 
gardless of the amount of super- 
vision. 

He can be depended upon to "get 
the job done" with a minimum of 
supervision, 

Ratee 's inability to satisfactorily 
discharge his financial obligations 
shows a lack of responsibility. 

Additional Indexing Clues : 

Always ready to accept additional 
responsibility) 

Willing to assume or accept added 
responsibility 

Works well on his own 

Completely reliable 

Job done without supervision 

Can work without direction 

Requires no supervision 

Dependable 



RELIABILITY AND DEPENDABILITY 4 

CONDUCT, INTEGRITY, AND PRIDE 3 
RELIABILITY AND DEPENDABILITY 4 

RELIABILITY AND DEPENDABILITY 3 

CONDUCT, INTEGRITY, AND PRIDE 2 
RELIABILITY AND DEPENDABILITY 2 

RELIABILITY AND DEPENDABILITY 5 

RELIABILITY AND DEPENDABILITY 4 

RELIABILITY AND DEPENDABILITY 4 
RELIABILITY AND DEPENDABILITY 4 
RELIABILITY AND DEPENDABILITY 3 
RELIABILITY AND DEPENDABILITY 3 
RELIABILITY AND DEPENDABILITY 3 
RELIABILITY AND DEPENDABILITY 3 
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RELIABILITY AND DEPENDABILITY (Continued) 



Stable 


RELIABILITY 


AND 


DEPENDABILITY 


3 


Trustworthy 


RELIABILITY 


AND 


DEPENDABILITY 


3 


Responsible 


RELIABILITY 


AND 


DEPENDABILITY 


3 


Constancy 


RELIABILITY 


AND 


DEPENDABILITY 


3 


Does not need to be supervised 


RELIABILITY 


AND 


DEPENDABILITY 


3 


Performs without supervision 


RELIABILITY 


AND 


DEPENDABILITY 


3 


Accepted responsibilities and 
authority 


RELIABILITY 


AND 


DEPENDABILITY 


3 


Assumes added responsibilities 


RELIABILITY 


AND 


DEPENDABILITY 


3 


Mature 


RELIABILITY 


AND 


DEPENDABILITY 


3 


Reluctance to assume new tasks 


RELIABILITY 


AND 


DEPENDABILITY 


2 


Need for direction and checkup 
by superiors 


RELIABILITY 


AND 


DEPENDABILITY 


2 


Relinquishes responsibility 


RELIABILITY 


AND 


DEPENDABILITY 


2 



It is important to remember when indexing that the concept of responsibili- 
ty belongs under RELIABILITY AND DEPENDABILITY. 
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REPRESENTATION is the creating of an image of an organization to the external 
or internal environment, 

EXAMPLES : 



He is an outstanding representative 
of the Navy in all respects, 

Ratee has been very successful in 
improving his Branch Station's 
relations with local high schools 
and youth organizations. 

He takes an active part in church 
and civic programs, helping to 
uphold the Navy image in the 
community. 

The appearance of this building is 
a fine example and reflection of 
the Navy to the visiting civilians. 

Additional Indexing Clues : 

Active in extracurricular activities 

Takes part in extracurricular 
activities 

Public relations for his branch 
Status of his branch or job area 



REPRESENTATION 5 



REPRESENTATION 4 



REPRESENTATION 4 



REPRESENTATION 4 



REPRESENTATION 4 
REPRESENTATION 3 

REPRESENTATION 3 
REPRESENTATION 3 



Remember that if a ratee 's job title or main job duty is public relations, 
then the term PRODUCTIVITY AND ACHIEVEMENT would be used rather .than 
REPRESENTATION since the statement would be referring to the performance 
of his total job function. 

Some management" experts do not consider REPRESENTATION to be a separate 
managerial function. However, in this dictionary it is considered to be 
a separate function because of the importance given to relating to the 
civilian community by Naval personnel. 

If a ratee sets up a liaison with another organizational unit or division 
within the Navy. or an outside organization, the index term ORGANIZATION 
would be used. However, if a ratee uses the liaison or organizational 
structure that is already set up to enhance his division's working rela- 
tionship with other organizational units, the index term REPRESENTATION 
would be used. If a liaison is used as a channel of communication, then 
the index term USE OF COMMUNICATION would be used. The rapport that a 
ratee has with other organizational units also enhances his division's 
work, and the index term REPRESENTATION would be used. The rapport that 
a ratee has with his divisional personnel would be indexed with LEADER- 
SHIP AND DIRECTING, 
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REPUTE is the position or standing of an 
subordinates, peers, and superiors. 

EXAMPLES: 

He possesses the character, person- 
ality, and desire that commands 
the highest respect and admiration 
from his associates. 

This action has generated the 
highest degree of trust and con- 
fidence with his superiors. 

He readily obtains the confidence 
of all who come in contact with him. 

He is well liked and highly re- 
spected by juniors and seniors 
alike, and his easy-going manner 
and pleasant congeniality make 
him at home in any surroundings. 

Ratee is highly regarded by all 
of his men. 

He expresses himself clearly and 
logically and his views are re- 
spected by those with whom he works. 

Additional Indexing Clues : 

He has their full respect and gets 
it. 



individual in his relationships to 



CONDUCT, INTEGRITY, AND PRIDE 3 
CONDUCT, INTEGRITY, AND PRIDE 3 
DRIVE 3 
REPUTE 5 

REPUTE 5 



REPUTE 4 



REPUTE 4 

CONDUCT, ^INTEGRITY, AND PRIDE 3 
COOPERATION 4 
FLEXIBILITY 3 



REPUTE 4 



COMMUNICATION 3 
REPUTE 3 



REPUTE 4 
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He is highly regarded by his men. 
Gains the genuine respect 
Praised 

Inspires respect 

Justified others 1 confidence in him 

Commands respect and gets it 

Expertise widely acknowledged 
and respected by others 

His advice is sought by others 
because of his technical skill. 



REPUTE 4 
REPUTE 4 
REPUTE 4 
REPUTE 3 
REPUTE 3 
REPUTE 3 

TECHNICAL SKILLS 4 
REPUTE 3 

REPUTE 3 

TECHNICAL SKILLS 3 
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REPUTE (Continued) 



Merits the respect 



REPUTE 3 



Generates respect 



REPUTE 3 



Respected by 



REPUTE 3 



Popular 



REPUTE 3 



Many of the feelings that a ratee's peers and superiors have for him 
would be indexed as REPUTE (e.g., being liked or trusted by his superi- 
ors). However, if it is the ratee T s subordinates who have, these same 
feelings, it would indicate the ratee f s leadership abilities and be la- 
beled as such (e.g., being liked or trusted by his men). If the words 
respect or regard are used, then automatically label this statement with 
REPUTE, For example, "His concern for his men has won the respect of 
his subordinates." LEADERSHIP AND DIRECTING 3; REPUTE 3, 

Thp concept of a ratee T s repute may appear in the narrative text under 
the f$uise of one or mor° descriptive phrases, but this segment of text 
should be treated as containing a single concept and be labeled only once 
with REPUTE, For example, "He is well liked and highly respected," would 
be treated as one phrase or concept and would be indexed only once with 
the index term REPUTE, 
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RESOURCEFULNESS is the innovative or creative ability to make effective and 
efficient use of men and materials resulting in improved performance. 

EXAMPLES : 



He has consistently demonstrated 
exceptional resourcefulness in 
dealing with especially complicated 
projects. 

He makes the best use of all the 
resources at his command. 



RESOURCEFULNESS 5 



RESOURCEFULNESS 5 



A direct indication of the Ratee's 
resourcefulness is indicated in 
the fact that he insists on obtain- 
ing maximum utilization of the 
Navy*s training material. 

Since the turnover of these modules 
is high, this represents a signifi- 
cant savings to the Navy r s resources, 

He has demonstrated a remarkable 
ability for substituting compo- 
nents or test equipment that might 
be temporarily unavailable. 

In his area of responsibility he 
requires that consumable modules 
be repaired and reused. 

Chief XX is a resourceful man and 
his potential is great to become 
an even better Chief. 



RESOURCEFULNESS 3 
RESOURCEFULNESS 5 



RESOURCEFULNESS 4 



RESOURCEFULNESS 4 



RESOURCEFULNESS 3 



RESOURCEFULNESS 3 
POTENTIAL 4 



His imagination allows him to find 
new and different solutions to 
problems which others do not seem 
to be able to solve. 



RESOURCEFULNESS 3 
INTELLECTUAL FUNCTIONING 4 



Additional Indexing Clues : 

Always resourceful 
Fully utilizes his men 
Inventiveness 
Imagination 
Ingenuity 



RESOURCEFULNESS 4 
RESOURCEFULNESS 4 
RESOURCEFULNESS 3 
RESOURCEFULNESS 3 
RESOURCEFULNESS 3 
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RESPONSIVENESS is the quality of reacting promptly, readily, accurately, and 
alertly to suggestion, instruction, or orders. 

EXAMPLES : 



Ratee is extremely cooperative in 
all his undertakings when given 
any task, and completes it in the 
most expeditious manner. 



COOPERATION 5 
RESPONSIVENESS 5 



He is quick to respond to any 
situation or problem and to find 
a solution. 



RESPONSIVENESS 4 
INTELLECTUAL FUNCTIONING 3 



He uncomplainingly responds when 
called upon to meet unscheduled 
commitments, frequently under 
adverse conditions. 



RESPONSIVENESS 4 
ENDURANCE 3 



He is responsive and agreeable to 
demands upon him and constantly 
seeks ways of improving working 
conditions and morale. 



RESPONSIVENESS 3 
COOPERATION 3 
LEADERSHIP AND DIRECTING 



Additional Indexing Clues : 

Alertness is usually considered to be a part of intellectual functioning 
However, if a statement refers to a ratee 1 s alertness to an order, then 
consider it to be a responsive gesture on the part of the ratee. 

When in doubt about whether a statement constitutes a cooperative or re- 
sponsible attitude as opposed to responsiveness on the part of the ratee 
use either COOPERATION or RELIABILITY AND DEPENDABILITY rather than RE- 
SPONSIVENESS. 

Always use the index term RESPONSIVENESS whenever the word responds is 
used in a statement. 

The time element is an important clue to the use of this term.and, there 
fore, would be used to label the following examples: 



In a minimum of time 



RESPONSIVENESS 5 



He carries out all assignments 
expediently and efficiently. 

He meets his responsibilities 
and quotas in a timely manner. 



RESPONSIVENESS 4 

PRODUCTIVITY AND ACHIEVEMENT 4 

RESPONSIVENESS 3 



Ratee presented material to the RESPONSIVENESS 3 

students in an expeditious manner. 
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SERVICE MOTIVATION refers to a serviceman's contentment with life in the Navy, 
or to his desire or lack of desire to fulfill his commitment to his job 
duties in the Navy. 



EXAMPLES : 

He is extremely dedicated to his 
division, department, and ship. 

He is an active Navy promoter and 
is the first to step forward to 
educate subordinates on Navy policy, 
procedures, or benefits. 

Ratee always wears his uniform with 
pride, both on and off the ship. 

He exhibits pride in being part of 
this command and is a definite 
asset to the Naval service. 

He is a dedicated career man who 
displays pride in the Navy and the 
Squadron. 

Additional Indexing Clues : 

The ratee always has the best 
interests of the Navy in mind. 

Pride in the Navy 

Pride in his unit 

Dedication or devotion to duty 

Loyal to duty and men 



SERVICE MOTIVATION 5 



SERVICE MOTIVATION 4 
SERVICE MOTIVATION 4 



SERVICE MOTIVATION 4 



SERVICE MOTIVATION 3 
ASSET TO THE NAVY 4 



SERVICE MOTIVATION 3 
SERVICE MOTIVATION 3 



SERVICE MOTIVATION 5 

SERVICE MOTIVATION 3 

SERVICE MOTIVATION 3 

SERVICE MOTIVATION 3 

SERVICE MOTIVATION 3 



Enjoying his work 



SERVICE MOTIVATION 3 
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SKILLS AND ABILITIES are those qualities that influence the producing of a 
desired result in an efficient and effective manner. 



EXAMPLES: 

He is able to direct the efforts of 
Line Personnel in an efficient and 
effective manner; this is reflected 
in the Ratee by a multiple of ex- 
ceptional qualities. 

This man is extremely competent. 

Above average in all areas, he is 
fully qualified to assume the 
greater responsibility of the next 
higher rank. 

He daily demonstrates all of the 
highly desirable traits of a Chief 
Petty Officer. 

Ratee is very meticulous and 
thorough. 

His natural abilities and responsi- 
ble approach to recruiting have 
enabled the ratee to outperform 
his contemporaries. 

Additional Indexing Clues : 

Exceptionally well qualified 

Outstanding individual 

Well-qualified Supervisor 

Efficient 



LEADERSHIP AND DIRECTING 4 
SKILLS AND ABILITIES 5 



SKILLS AND ABILITIES 5 

SKILLS AND ABILITIES 4 
SKILLS AND ABILITIES 4 
POTENTIAL 3 



SKILLS AND ABILITIES 4 



SKILLS AND ABILITIES 4 
SKILLS AND ABILITIES 4 

SKILLS AND ABILITIES 3 
RELIABILITY AND DEPENDABILITY 3 
PRODUCTIVITY AND ACHIEVEMENT 4 



SKILLS AND ABILITIES 5 

SKILLS AND ABILITIES 5 

SKILLS AND ABILITIES 4 

SKILLS AND ABILITIES 4 



Any mention of managerial or administrative abilities would be labeled 
MANAGEMENT FUNCTIONS. 

SKILLS AND ABILITIES is used as a catchall term for those skills and 
abilities which do not fit any of the more specific index terms that 
come under SKILLS AND ABILITIES in the hierarchy. The following terms 
would be labeled under SKILLS AND ABILITIES, since there is no other more 
specific index term that would describe them: qualities, skills, traits, 
attention to detail, meticulous, thorough, methodical, competent, capable 
decisiveness, deliberate in thinking (deliberate in action would be index 
ed as CONDUCT, INTEGRITY, AND PRIDE), and talented. 
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STAFFING is the manning of and keeping manned the positions provided for by 
the organization structure. Includes training and the appraisal or 
assessment of personnel. 

EXAMPLES: 



His instructions are presented in 
such a manner that maximum training 
is accomplished in the time alloted. 

He spent a great deal of extra time 
with his men and turned out well 
trained, well motivated men for the 
fleet. 



COMMUNICATION 3 
STAFFING 5 



LEADERSHIP AND DIRECTING 4 
STAFFING 4 

LEADERSHIP AND DIRECTING 4 



He is continually researching the 
available training and ensuring 
that assigned personnel have what 
is needed. 



STAFFING 4 
CONTROLLING 4 



Ratee has tutored division person- STAFFING 3 

nel in all aspects of the Navy 
publications system. 

Additional Indexing Clues : 

Men re-enlist STAFFING 3 



The manning of and keeping manned the job positions would involve selec- 
tion, training, and the compensation of personnel in order to keep them 
in the Navy. It is important to remember that if the main job duty is 
recruiting, then usually a statement about manning of and keeping manned 
positions in the Navy would be the result of a ratee' s total management 
function, and hence would be indexed as PRODUCTIVITY AND ACHIEVEMENT. 
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TECHNICAL SKILLS refer to the understanding and demonstration of techniques 
which one applies to a task. 

EXAMPLES : : 



The ratee has an outstanding 
knowledge of all Electrical Drone 
Systems in the squadron aircraft. 



TECHNICAL SKILLS 5 



BTC has an excellent working and 
practical knowledge of the PMS 
system but has a tendency to be 
lax in the administrative phase 
-:f the system. 



TECHNICAL SKILLS 5 
MANAGEMENT FUNCTIONS 2 



He is well versed in the 3-M System 
and always exhibits sound manage- 
ment practices. 



TECHNICAL SKILLS 4 
MANAGEMENT FUNCTIONS 4 



Ratee single-handedly tore down 
and rebuilt an engine in 16 hours 
so the ship would be ready for the 
cruise. 



TECHNICAL SKILLS 4 



He is a very knowledgeable techni- 
cian who attempts to keep constantly 
abreast of current changes by de- 
voting his off duty time to the 
study of technical and nontechnical 
material through. evening ( college 
courses. 



TECHNICAL SKILLS 4 
DRIVE 5 



His technical competence and re- 
sourcefulness contribute to ships 
in tending to maintain operational 
readiness . 



TECHNICAL SKILLS 3 
RESOURCEFULNESS 3 
PRODUCTIVITY AND ACHIEVEMENT 3 



Chief XX was relieved of K r s duties 
as the ship T s Oil King aft^r serving 
in the capacity for approximately 
two months. He was removed from 
this billet because of his lack of 
professional knowledge and technical 
know-how in the art of refueling. 



AWARDS AND PUNISHMENT 1 
PROFESSIONALISM 2 
TECHNICAL SKILLS 2 



Additional Indexing Clues : 
Expertise widely acknowledged 



TECHNICAL SKILLS 4 
REPUTE 4 



Professional knowledge 



PROFESSIONALISM 3 
TECHNICAL SKILLS 4 
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TECHNICAL SKILLS (Continued) 

The following would be indexed as TECHNICAL SKILLS 3; background, know- 
ledge of the responsibility of the rate, technical skills, technical 
knowledge, understanding of job duties, experience, understanding of 
work, "know-how. " 

The technical skills that a ratee possesses may be referred to by the 
evaluator with one or more descriptive phrases, but this part of the sen- 
tence should be treated as a single entity and labeled only once with 
TECHNICAL SKILLS. For example, "...his lack of professional knowledge 
and technical know-how in the art of refueling...." PROFESSIONALISM 2; 
TECHNICAL SKILLS 2. 

If the skill or knowledge is in managing or in one of the specific mana- 
gerial functions, just use MANAGEMENT FUNCTIONS or the specific function 
as the label. For example, "He is skilled at managing his men," would be 
indexed as LEADERSHIP AND DIRECTING 3. 

In the original version of this dictionary there was a term FUND OF KNOW- 
LEDGE which was replaced by TECHNICAL SKILLS since managers at a junior 
level have technical as well as managerial functions. This is the reason 
why such phrases as "understanding of job duties, experience, and under- 
standing of work" are labeled as TECHNICAL SKILLS. It is important to 
keep this term in mind and to discriminate between managerial functions 
and productivity or achievement as opposed to the demonstration of a 
technical skill. 

When a ratee is in the process of acquiring new knowledge or a new skill, 
it would be an indication of his drive and be labeled as such. The iu- 
dexer should not use TECHNICAL SKILLS unless .the ratee has finished the 
course and, therefore, already has the knowledge or skill. 
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USE OF COMMUNICATION is the use an individual makes of the opportunities for 
. information exchange within an organization. 

SEE ALSO: COMMUNICATION 

EXAMPLES : 



He is an excellent administrator 
and a skillful supervisor, and he 
keeps, himself and his superiors 
fully inforuied of all facets of 
his branch's operation. 

He always keeps his superiors 
informed of any problems and the 
status of work in progress. 

Ratee always consults with his 
Division Officer concerning his 
wishes on a matter, unless time 
is essential. 



MANAGEMENT FUNCTIONS 5 
LEADERSHIP AND DIRECTING 4 
USE OF COMMUNICATION 5 



USE OF COMMUNICATION 5 



USE OF COMMUNICATION 4 



He keeps his seniors alerted to 
pending problems. 

Additional Indexing Clues : 

Keeps superior well infor 

Uses communication to arouse inter- 
est, convince, and produce desired 
results 



USE OF COMMUNICATION 3 



USE OF COMMUNICATION 4 
USE OF COMMUNICATION 3 



This term is used not to indicate how a ratee communicates, but whether 
or not he uses his ability to communicate for the purpose of information 
exchange . 

If a ratee sets up a liaison with another organizational unit or division 
within the Navy or an outside organization, the index term ORGANIZATION 
would be used. However, if a ratee uses the liaison or organizational 
structure that is already set up to enhance his division's working rela- 
tionship with other organizational units, the index term REPRESENTATION 
would be used. If a liaison is used as a channel of communication, then 
the index term USE OF COMMUNICATION would be used. 
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APPENDIX B 



OUTPUT FROM PROGRAM BMD01D - SIMPLE DATA DESCRIPTION 
FOR THE CROSS VALIDATION AND GENERALIZATION SAMPLES 



The output of this computer program lists the 
arithmetic mean, the standard deviation, the standard 
error of the mean, the maximum value, the minimum 
value, and the range for each of the 67 variables 
derived from the content analysis • The output is 
presented in six parts for each of the two samples. 
First is shown the output for the Upper, Middle, and 
Lower criterion groups on the evaluation section of 
the Evaluation Report. This is followed by the output 
for the Upper, Middle, and Lower criterion groups on 
the justification section of the Evaluation Report. 
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CROSS VALIDATION SAMPLE (AT 1 s and BT T s) 
UPPER CRITERION GROUP (N=74) 
1 9 R- E V ALU AT ION SECTION 



No. of 

Variable Mean S.D. 



1 


0. 


,3649 


0. 


,5382 


2 


0. 


.1351 


0. 


.5054 


3 


0. 


.5946 


0. 


,9920 


4 


0. 


,0811 


0. 


,2748 


5 


0. 


.0541 


0. 


,2277 


6 


0. 


.0811 


0. 


,3208 


7 


0. 


,0676 


0. 


,2527 


8 


0. 


,0135 


0. 


,1162 


9 


0. 


,4459 


0. 


,6650 


10 


0. 


,1757 


0. 


,4490 


11 


0. 


6486 


1. 


,1157 


12 


0. 


,0946 


0. 


,2947 


13 


0. 


1216 


0. 


3291 


14 


0. 


0541 


0. 


,2277 


15 


0. 


0811 


0. 


3208 


16 


0. 


,1486 


0. 


,3946 


17 


0. 


2297 


0. 


6092 


18 


0. 


2838 


0. 


5618 


19 


0. 


2973 


0. 


6133 


20 


0. 


0676 


0. 


2527 


21 


0.0811 


0. 


4898 


22 


o. 


6216 


0. 


9320 


23 


1. 


,3108 


1. 


,2488 


24 


0. 


2297 


0. 


4839 


25 


1. 


1622 


1. 


4807 


26 


0.2297 


0. 


5376 


27 


0. 


3649 


0. 


5631 


28 


0. 


2297 


0. 


6092 


29 


0. 


0676 


0. 


2527 


30 


8. 


,2027 


5. 


,4446 


31 


10. 


,9594 


1. 


,4471 


32 


10. 


,2838 


1. 


,0793 


33 


11. 


.2838 


2. 


1677 


34 


10. 


,1757 


0. 


6050 


35 


10.0676 


0.3021 


36 


10. 


1757 


0. 


6896 


37 


10. 


.1757 


0.6896 


38 


10.0135 


0.1162 


39 


10.8243 


1. 


2091 


40 


10.3919 


L. 


,0179 


41 


11. 


.3378 


2. 


6394 


42 


10. 


,2027 


0. 


,6617 


43 


10.2027 


0. 


, 6189 



S.E. of 



Mean 


Max. 


Min.. 


Rangi 


0.0626 


2 


0 


2 


0.0588 


3 


0 


3 


0.1153 


4 


0 


4 


0.0319 


1 


0 


1 


0.0265 


1 


0 


1 


0.0373 


2 


0 


2 


0.0294 


1 


0 


1 


0.0135 


1 


0 


1 


0.0773 


3 


0 


3 


0.0522 


2 


0 


2 


0.1297 


5 


0 


5 


0.0343 


1 


0 


1 


0.0383 


1 


0 


1 


0.0265 


1 


0 


1 


0.0373 


2 


0 


2 


0.0459 


2 


0 


2 


0.0708 


3 


0 


3 


0.0653 


3 


0 


3 


0.0713 


3 


0 


3 


0.0294 


1 


0 


1 


0.0569 


4 


0 


4 


0.1083 


3 


0 


3 


0.1452 


4 


0 


. 4 


0.0563 


2 


0 


2 


0.1721 


6 


0 


6 


0.0625 


3 


0 


3 


0.0655 


2 


0 


2 


0.0708 


3 


0 


3 


0.0294 


1 


0 


1 


0.6329 


23 


0 


23 


0.1682 


15 


10 


5 


0.1255 


17 


10 


7 


0.2520 


19 


10 


9 


0.0703 


13 


10 


3 


0.0351 


12 


10 


2 


0.0802 


14 


10 


4 


0.0802 


14 


10 


4 


0.0135 


11 


10 


1 


0.1406 


14 


10 


4 


0.1183 


15 


10 


5 


0.3068 


23 


10 


13 


0.0769 


13 


10 


3 


0.0719 


13 


10 


3 
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CROSS VALIDATION SAMPLE (AT's and BT's) 
UPPER CRITERION GROUP (N=74) 
19R-EVALUATI0N SECTION 
(Continued) 



No. S.E. of 



Vfl T*"? flT^ 1 p 


llCci Ll 


D • U . 


Mean 


Max . 


Min . 


Range 


44 




D AA^9 


U . UjJ, j 


1:5 


10 


3 


45 




O Q 7 R R 
v» O / JJ 


n i m a 


Id 


ID 


r 
0 


46 


X v • <J -/ <J 


fl 8H9 7 
U • OK)/. 1 


U • U/ j j 


13 


ID 


3 


47 


in A^9A 


1 91 79 
J. • £ J. / £ 


U. 


15 


10 


5 


A£ 

H O 


-LU • 3 uuu 


1 . Uoo3 


U. 126)5 


16 


10 


6 


AQ 


-LU • U£_LO 


i /. q n £ 


U. loo 3 


18 


10 


8 




-LU • _L£_LO 


fl R 9 9 9 


U. UoU/ 


13 


10 


3 


si 


_LU • _L £_LO 


n aqi r 


u . u/ 


15 


10 


5 


52 




9 1 7 R9 


fl 9 ^ 9Q 


iy 


10 


y 


53 


13.0811 


2.9319 


0.3408 


21 


10 


ll 


54 


10.6892 


1.4517 


0.1688 


16 


10 


6 


55 


12.1621 


3.0611 


0.3558 


25 


10 


" 15 


56 


10.4594 


1.1842 


0.1377 


17 


10 


7 


57 


10.6216 


1.0028 


0.1166 


14 


10 


4 


58 


10.5000 


1.3574 


0.1578 


17 


10 


7 


59 


10.1351 


0.5570 


0.0648 


13 


10 


3 


60 


306.1311 


17.0533 


1.9824 


336 


201 


135 


6.1 


2.7432 


2.4942 


0.2899 


11 


0 


11 


62 


3.6486 


2.7718 


0.3222 


12 


0 


12 


63 


1.9459 


2.0197 


0.2348 


7 


0 


7 


64 


0.0000 


0.0000 


'0.0000 


0 


0 


0 


65 


0.0000 


0.0000 


0.0000 


0 


0 


0 


66 


78.3376 


49.0724 


5.7045 


212 


0 


212 


67 


5.6081 


3.3508 


0.3895 


14 


0 


14 



\ 



ERIC 



154 



CROSS VALIDATION SAMPLE (AT's and BT T s) 
MIDDLE CRITERION GROUP (N=74) 
19R- EVALUATION SECTION 



No. of ■ • S.E. of 



riable 


ile an 


S.D. 


Mean 


Max. 


Min. 


Rangi 


1 


0.3108 


0.5471 


0.0636 


2 


0 


2 


2 


0.2162 


0.5040 


0.0586 


2 


0 


2 


3 


1.0000 


1.2928 


0.1503 


4 


0 . 


4 


4 


0.2297 


0.5376 


0.0625 


2 


0 


2 


5 


0.1216 


0.3291 


0.0383 


1 


0 


1 


6 


0.0811 


0.2748 


0.0319 


1 


0 


1 


7 


0.0676 


0.3021 


0.0351 


2 


0 


2 


8 


0.0405 


0.1986 


0.0231 


1 


0 


1 


9 


0.3919 


0.7552 


0. 0878 


4 


0 


4 


10 


0.3108 


0.5951 


0.0692 


3 


0 


3 


11 


1.1892 


1.9492 


0.2266 


9 


0 


9 


12 


■0.4595 


0.7437 


0.0865 


4 


0 


4 


13 


0.1081 


0.3126 


0.0363 


1 


0 


1 


14 


0.0946 


0.3380 


0.0393 


2 


0 


2 


15 


0.2297 


0.4235 


0.0492 


1 


0 


1 


16 


0.2162 ' 


0.4145 


0.0482 


1 


0 


1 


17 


0.3649 


0.7134 


0.0829 


3 


0 


3 


18 


0.2162 


0.4463 


0. 0519 


2 


0 


2 


19 


0.3514 


0.6066 


0.0705 


2 


0 


2 


20 


0.2568 


0.5748 


0.0668 


3 


0 


3 


21 


0.0405 


0.1986 


0.0231 


1 


0 


1 


22 


0.7162 


0.7855 


0.0913 


3 


0 


3 


23 


0.9189 


0.9029 


0.1050 


3 


0 


3 


24 


0.0946 


0.2947 


0.0,343 


1 


0 


1 


25 


1.0135 


1.1526 


0.1340 


5 


0 


5 


26 


0.3378 


0.5558 


0.0646 


2 


0 


2 


27 


0.4459 , 


0.6853 


0.0797 




0 


3 


28 


0.2432 


0.5442 


0.0633 


2 


0 


2 


29 


* 0.1486 


0.3582 


0.0416 , 


1 


0 


1 


30 


10.0811 


5.6052 


0.6516 


25 


0 


25 


31 


10.7027 


1.2576 


0.1462 


15 


10 


5 


32 


10.4054 


0.9497 


0.1104 


14 


10 


4 


33 


11.9189 


2.6881 


0.3125 


19 


10 


9 


34 


10.4054 


0.9640 


0.1121 


14 


10 


4 


35 


10.2297 


0.6313 


0.0734 


12 


10 


2 


36 


10.1486 


0.5150 


0.0599 


12 


10 


2 


37 


10.1351 


0.6688 


0.0777 


15 


10 


5 


38 


10.0811 


0.4303 
1.5641 


0.0500 


13 


10 


3 


39 


10.7297 


0.1818 


20 


10 


10 


40 


10.6081 


1.1799 


0.1372 


16 


10 


6 


41 


11.6351 


2.8649 


0.3330 


22 


5 


17 


42 


* 10.8919 


1.7405 


0.2023 


20 


10 


10 


43 


10.2432 


0.7367 


0.0856 


13 


10 


3 
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CROSS VALIDATION SAMPLE (AT's and BT's) 
MIDDLE CRITERION GROUP (N=74) 
19R-EVALUATI0N SECTION 
(Continued) 



No. of 
Variable Mean 



44 


10.1351 


45 


10.3919 


46 


10.2838 


47 


10.6621 


48 


10.4324 


49 


10.4459 


50 


10.4054 


51 


10.0405 


52 


11.2973 


53 


12.0270 


54 


10.2838 


55 


11.8243 


56 


10.5540 


57 


10.8784 


58 


10.4054 


59 


10.2703 


60 


308.4282 


61 


2.1216 


62 


4.3784 


63 


3.5270 


64 


0.1892 


65 


0.0000 


66 


86.5808 


67 


7.1621 



S.E. of/ 
S . D . Mean 



0.5811 0.0675 

0.7907 0.0919 

0.6086 0.0708 

1.3677 0.1590 

1.0347 0.1203 

0.9529 0.1108 

0.9054 0.1052 

0.1986 0.0231 

1.5055 0.1750 

2.0672 0.2403 

0.8840 0.1028 

2.1155 0.2459 

0.9951 0.1157 

1.4893 0.1731 

0.9351 0.1087 

0.7639 0.0888 

10.3687 1.2053 

2.0604 0.2395 

3.1740 0.3690 

2.7259 0.3169 

1.0813 ' 0.1257 

0.0000 0.0000 

42.8689 4.9834 

3.2011 0.3721 



Max. Min . Range 



13 


9 


4 


13 


10 


3 


13 


10 


3 


17 


10 


7 


16 


10 


6 


14 


8 


6 


13 


10 


3 


11 


10 


1 


17 


10 


7 


17 


10 


7 


13 


10 


3 


20 


10 


10 


14 


10 


4 


18 


10 


8 


14 


10 


4 


13 


9 


4 


337 


289 


48 


12 


0 


12 


13 


0 


13 


11 


0 


11 


9 


0 


• 9 


0 


0 


0 


214 


0 


214 


15 


0 


15 
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CROSS VALIDATION SAMPLE (AT's and BT's) 
LOWER CRITERION GROUP (N=74) 
19R-EVALUATI0N SECTION 



No. of • 

Variable Mean S.D. 



1 

1 


U . 


0 "7 n *3 

Z / U j 


U . 


^ n/. "7 


Z 


0 . 


1 o o o 

1892 


r\ 
U . 


/ c o c 

4586 


3 


(J . 


7973 


1 . 


on/ o 

, zU48 


4 


0 . 


2162 


r\ 
U . 


r o a/ 

5304 


5 


0 . 


1216 


r\ 
0 . 


3291 


6 


0 . 


1081 


0 . 


3906 


7 


0. 


1081 


0. 


3537 


8 


0 . 


r\ o "7 r\ 

0270 


0 . 


1633 


9 


0 . 


3108 


0. 


5471 


10 


0 . 


2297 


0. 


4547 


11 


1. 


0946 


1 . 


5891 


12 


0. 


2027 


0. 


,4048 


13 


0. 


0946 


0 . 


,2947 


1-4 


0. 


0541 


0. 


2277 


15 


0. 


2027 


0. 


,4373 


1.6 


0. 


2297 


0. 


5114 


17 


0. 


1757 


0. 


5063 


18 


0. 


0676 


0. 


2527 


19 


0. 


4730 


0. 


6667 


20 


0. 


1081 


0. 


3906 


21 


0. 


1622 


0. 


,4388 


22 


0. 


9459 


1. 


3838 


23 


1. 


3513 


1. 


1033 


24 


0. 


0541 


0. 


2815 


25 


0. 


9189 


1. 


2688 


26 


0. 


1757 


0. 


4785 


27 


0. 


5135 


0. 


6462 


28 


0. 


1757 


0. 


4174 


29 


0. 


1351 


0. 


3442 


30 


9. 


5135 


5. 


1398 


31 


10. 


4865 


1. 


0758 


32 


10. 


2703 


0.8648 


33 


11. 


0270 


1. 


8798 


34 


10. 


2027 


0. 


8273 


35 


10. 


2297 


0. 


,6733 


36 


10. 


,2027 


0. 


7211 


37 


10. 


,1216 


0. 


,6400 


38 


10.0270 


0. 


,1633 


39 


10.4189 


0. 


.8760 


40 


10. 


.3784 


0. 


.8712 


41 


11. 


.3243 


2. 


.4049 


42 


10.2973 


0. 


.6565 


43 


10. 


,1757 


0. 


.6050 



S.E. of 



Mean 


Max. 


Min. 


Rangi 


0. 0587 


2 


0 


2 


0. 0533 


2 


0 


2 


0.1401 


8 


0 


8 


0. 0617 


3 


0 


3 


0. 0383 


1 


0 


i 


0.0454 


2 


. 0 


2 


0. 0411 


2 


0 


2 


0.0190 


1 


0 


1 


0.0636 


2 


0 


2 


0.0529 


2 


0 


2 


0.1847 


7 


0 


7 


0.0471 


1 


0 


1 


0.0343 


1 


0 


1 


0.0265 


L 


0 


1 


0.0508 


2 


0 


2 


0.0595 


3 


0 


3 


0.0589 


• 2 


0 


2 


0.0294 


1 


0 


1 


0.0775 


3 


0 


3 


0.0454 


2 


0 


2 


0.0510 


2 


0 


2 


0.1609 


7 


0 


7 


0.1283 


5 


0 


5 


0.0327 


2 


0 


2 


0.1475 


5 


0 


5 


0.0556 


2 


0 


2 


0.0751 


2 


0 


2 


0.0485 


2 


0 


2 


0.0400 


1 


0 


1 


0.5975 


26 


0 


26 


0.1251 


14 


9 


5 


0.1005 


.14 


9 


5 


0.2185 


19 


8 


11 


0.0962 


13 


7 


6 


0.0783 


13 


10 


3 


0.0838 


14 


10 


4 


0.0744 


14 


9 


5 


0.0190 


11 


10 


1 


0.1018 


14 


9 


5 


0.1013 


13 


9 


4 


0.2796 


25 


9 


16 


0.0763 


13 


10 


3 


0.0703 


13 


10 


3 



0 

ERJ.C 
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CROSS VALIDATION SAMPLE (AT's and BT's) 
LOWER CRITERION GROUP (N=74) 
1 9 R- EV ALUAT ION SECTION 
(Continued) 



No. of 


> 




S.E. of 








Variable 


Mean 


S.D. 


Mean 


Max. 


Min. 


Range 


44 


10.0405 


0.3070 


0.0357 


12 


9 


3 


45 


10.2432 


0.6582 


0.0765 


13 


9 


4 


46 


10.2162 


0.8321 


0.0967 


15 


9 


6 


47 ' 


10.3108 


0.9352 


0.1087 


15 


10 


5 


48 


10.1081 


0.4553 


0.0529 


13 


10 


3 


49 


10.5676 


1.0609 


0.1233 


14 


7 


7 


50 


10.1216 


0.6611 


0.0769 


15 


9 


6 


51 


10.2973 


0.8231 


0.0957 


14 


10 


4 


52 


11.6621 


2.3366 


0.2716 


20 


10 


10 


53 


12.2973 


2.2618 


0.2629 


18 


9 


. 9 


54 


10.0946 


0.8137 


0.0946 


16 


8 


8 


55 


11.3919 


2.1696 


0.2522 


.19 


8 


11 


56 


10.3108 


0.9641 


0.1121 


15 


10 


5 


57 


10.7973 


1.3546 


0.1575 


15 


9 


6 


58 


10.2027 


0.7018 


0.0816 


14 


9 


5 


59 


10.2297 


0.6313 


0.0734 


13 


10 


3 


60 


304.0640 


8.6142 


1.0014 


329 


290 


39 


61 


1.1351 


1.4835 


0.1724 


6 


0 


6 


62 


4.2432 


2.7339 


0.3178 


11 


0 


11 


63 


3.1621 


2.5537 


0.2969 


13 


0 


13 


64 


0.9595 


2.0703 


0.2407 


12 


0 


12 


65 


0.0135 


0.1162 


0.0135 


1 


0 


1 


66 


87.1889 


46.2204 


5.3730 


287 


0 


287 


67 


6.5135 


2.9759 


0.3459 


16 


0 


16 



9 

ERIC 
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CROSS VALIDATION SAMPLE (AT's and BT T s) 
UPPER CRITERION GROUP (N=74) 
19 S- JUSTIFICATION SECTION 



No. of S.E. of 



V ciricLD-L 6 


Mean 


C T\ 


Mean 


Max. 


Min. 


Range 
3 


1 


u . 443y 


U . / z4z 


U. 0842 


3 


0 


o 
z 


u . / / U J 


i q on o 


U. Ihlb 


o 

8 


0 


8 


o 
J 


i o /. e o 


1 none 


0.2243 


o 

8 


0 


8 




n 1 7C7 

U . 1 / 3 / 


H /. "7 Q C 

U . 4 /o3 


0. 0556 


3 


0 


3 


3 


U . 4U34 


U . 75/ 0 


0.0880 


3 


0 


3 


D 


U . IzlO 


r\ o ^ o /. 
U . 30o4 


0. 0428 


2 


0 


2 


/ 


U . Z3DO 


U . 7034 


0. 0818 


4 


0 


4 


O 
O 


n no/. ^ 

u.uy4o 


/.TIT 
U . 4111 


U. 0478 


3 


0 


3 


Q 

y 


n QQ7Q 

U . o 3 / o 


n r\ o n "7 
1 - U2U / 


0.1187 


4 


0 


4 


1U 


1 1 7C7 
1.1/3/ 


n o o "7 n 
1 . 3Z /9 


0. 1544 


5 


0 


5 


ii 


z . Uo / 0 


1 . 9oo3 


0.2312 


7 


0 


7 


1 9 
1Z 


U . 3341 


r\ o o n n 

0 . 8299 


0. 0965 


3 


0 


3 


1 9 
1 J 


n O Q 9 Q 
U-.ZOJO 


r* c o £ o 
U . 33 59 


0. 0624 


2 


0 


2 




U . KJzfH 0 


U . 3 / 03 


U. 0437 


2 


0 


2 


13 


n 7 /• ^ 9 

U . / 4 3 Z 


1 . lZ34 


n i on/: 
U. 13Ud 


5 


0 


5 


ID 


U • 0331 


U . 0333 


n n n n o 

U. 0992 


4 


0 


4 


1 7 
1 / 


n /• 9 9 /• 

U . 43Z4 


U . o / / 1 


n n n o n 

U. 1U2U 


5 


0 


5 


io 


n ^9 70 
U . 3Z / U 


n 7 O ^ 7 
U . / Z3 / 


U. 0844 


3 


0 


3 


1 Q 
1-7 


1 . uuuu 


i . loyo 


U. lZ3Z 


5 


0 


5 


zu 


n ^ ^7 a 

U . 3D / 0 


U . ool4 


U . 1UU1 


3 


0 


3 


Zl 


U . 3133 


C\ 7 £ O Q 

u . /ozy 


n n o o "7 

U. 0887 


4 


0 


4 


9 9 
Z Z 


i ^ o 7 n 

1 . 3z / U 


1 . /U39 


U. 1983 


8 


0 


8 


9 9 
Z J 


Z . 3D / D 


Z . 2639 


0.2632 


14 


0 


14 


9 A 
Z4 


U . lUol 


r\ o c o t 
U . 3337 


0. 0411 


2 


0 


2 


9 ^ 
Z3 


9 Ol AO 
Z . ZlDZ 


o o o o /. 

z . zzz4 


0.2583 


10 


0 


10 


9 £ 
ZD 


n Q 7 Q /. 
U . 3 /o4 


U . 0339 


0.0762 


3 


0 


3 


Z / 


l.lDZZ 


1 . UzU / 


0. 1187 


4 


0 


4 


OQ 
ZO 


n o £ / n 

0 . 3649 


0 . 6939 


0.0807 


3 


0 


3 


9 Q 

zy 


n /.ion 

u . 4loy 


U . bozo 


0. 0793 


3 


0 


3 


30 


22.3916 


14.7358 


J .7130 


91 


5 


86 


31 


11.0270 


1.7039 


0.1981 


18 


10 


8 


32 


11.3919 


2.3456 


0.2727 


23 


10 


13 


33 


13.9054 


3.8327 


0.4455 


28 


10 


18 


34 


10.2297 


0.6313 


0.0734 


13 


10 


3 


35 


10.8243 


1.5907 


0.1849 


18 


10 


8 


36 


10,2838 


0.8840 


0.1028 


15 


10 


5 


37 


10.4459 


1.2836 


0.1492 


18 


10 


8 


38 


10.1216 


0.4953 


0.0576 


13 


10 


3 


39 


11.5405 


2.0419 


0.2374 


17 


10 


7 


40 


12.4730 


2.7708 


0.3221 


20 


10 


10 


41 


13.8513 


3.6440 


0.4236 


23 


10 


13 


42 


11.1486 • 


1.8408 


0.2140 


18 


10 


8 


43 


10.5000 


1.1257 


0.1.309 


16 : 


10 


6 
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CROSS VALIDATION SAMPLE 
UPPER CRITERION GRl 
19S- JUSTIFICATION 



No. of 

Variable Mean S.D. 



44 


10.1757 .. 


0.7468 


45 


11.5811 


2.4214 


46 


11.0000 


1.5170 


47 


10.7162 


1.6346 


48 


10.9459 


1.3936 


49 


11.9594 


2.1672 


50 


10.9865 


1.5746 


51 


10.6892 


1.3742 


52 


13.1216 


3.4915 


53 


15.5811 


4.6435 


54 


10.3243 


1.0612 


55 


14.0946 


4.1814 


56 


10.5946 


1.0844 


57 


12.4730 


2.3827 


58 


10.5676 


1.0990 


59 


10.7703 


1.2985 


60 


333.3606 


27.1629 


61 


6.8648 


10.7707 


62 


9.9730 


8.5303 


63 


7.1351 


6.7221 


64 


0.0000 


0.0000 


65 


0.0000 


0.0000 


66 


201.4052 


146.3721 


67 


11.6081 ■ 


4.8224 



(AT's and BT's) 
UP (N=74) 
SECTION 



S.E. of 



Mean 


Max. 


. Min. 


Range 


0.0868 


15 


10 


5 


0.2815, 


20 


10 


10 . 


0.1764 


17 


10 


7 


0.1900 


19 


10 


9 


0.1620 


15 


10 


5 


0.2519 


18 


10 


8 


0.1830 


17 


10 


7 


0.1597 


20 


10 


10 


0.4059 


28 


10 


18 


0.5398 


39 


10 


29 


0.1234 


16 


10 


6 


0.4861 


28 


10 


18 


0.1261 


15 


10 


5 


0.2770 


19 


10 


y 


0.1278 


15 


10 


5 


0.1510 


15 


10 


5 


3.1576 


449 


299 


150 


1.2521 


91 


0 


91 


0.9916 


59 


0 


59 


0.7814 


44 


0 


44 


0.0000 


0 


0 


0 


0.0000 


0 


0 . 


0 


.17.0154 


896 


35 


861 


0.5606 


27 


4 


23 
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CROSS VALIDATION SAMPLE (AT's and BT's) 
MIDDLE CRITERION GRO,UP (N=74) 
19S- JUSTIFICATION SECTION 



No. of 
Variable Mean 



1 

-i- 


0 391 9 


2 


0. 5135 


3 


1 SS40 


A 
•■t 


0. 2162 


5 


0 3784 


6 


0.1622 


7 


0 1 ARfi 


8 




9 


0 391 9 




0 7973 


1 1 


2 . 0000 


12 


0,4595 


13 


0. 0946 


14 


. 0. 0811 


15 


0.6081 


16 


0.3919 


i 7 

X / 


0. 2568 


18 


0.2838 


19 


0.7432 


20 


0 4710 


Z. -L 


0 ?7 01 


Z. Z. 


0 741? 


ZO 


1 ARfiS 




0 OA OS 


z. 




26 


0.2162 


27 


0.7838 


28 


0.2703 


29 


0.1757 


30 


15.6890 


31 


11.0270 


32 


10.9594 


33 


12.8919 


34 


10.3919 


35 


10.6216 


36 


10.3784 


37 


10.2297 


38 


10.1216 


39 


10.6216 


40 


11.5946 


41 


13.6081 


42 


10.9324 


43 


10.2162 



9 

ERIC 



S.E. of 



S.D. 


Mean 


Max • 


Min ■ 


Rang* 


0. 5690 


0 . 0661 


2 


o 


2 


1. 0369 


0.1205 


7 


o 


7 


1 . 9525 


0.2270 


7 


o 


7 


0.5040 


0.0586 


2 


o 


2 


0. 8227 


0.0956 


4 


o 


4 


0. 5740 


0 . 0667 


4 


o 


4 


0.4588 


0 0S33 


3 


o 


3 
-> 


0. 3021 


0.0351 


2 


o 


2 


0.6583 


0. 0765 


2 


o 


2 


1. 3846 


0.1610 


9 


0 


9 

y 


2.1390 


0 ?4ft7 


9 


o 


9 


0. 7797 


0. 0906 


3 


o 


3 


0. 2947 


0.0343 


1 


o 


1 


0.2748 


0. 0319 


1 


o 


1 


0.8730 


0.1015 


4 


o 


4 


0.6153 


0. 0715 


2 


o 


2 


0* 5250 


0.0610 


2 


o 


2 


0.5618 


0.0653 


2 


o 


2 


0. 9801 


0.1139 


4 


o 


4 


1.1494 


0.1336 


9 


o 


9 


0.6259 


0. 0728 


3 


o 


3 
-> 


1. 0076 


0.1171 


5 


o 


5 


1.4358 


0.1669 


7 


o 


7 


0. 1986 


0 0911 


1 


n 


i 

j. 


1 .7354 


0 . 2017 


7 


o 


7 


0. 5304 


0 . 0617 


2 


o 


9 
z. 


0.9107 


0.1059 


3 


0 


.J 


0.6036 


0.0702 


3 


0 


3 


0.4174 


0.0485 


2 


0 


2 


9.5657 


1.1120 


44 


1 


43 


1.4709 


* J. 1710 


. 15 


10 


5 


1.9120 


0.2223 


23 


10 


13 


3.7106 


0.4314 


25 


10 


15 


0.9482 


0.1102 


14 


10 


. 4 


1.3619 


0.1583 


17 


10 


7 


1.3314 


0.1548 


19 


10 


9 


0.7503 


0.0872 


15 


10 


5 


0.5722 


0.0665 


14 


10 


4 


1.1312 . 


0.1315 


14 


10 


4 


2.6062 


0.3030 


23 


10 


13 


3.6067 


0.4193 


26 


10 


16 


1.7621 


0.2048 


19 


10 


9 


0.7266 


0.0845 


13 


10 


3 
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CROSS VALIDATION SAMPLE (AT's and BT T s) 
MIDDLE CRITERION GROUP (N=74) 
19S- JUSTIFICATION SECTION 
(Continued) 



No. of 

Variable Mean S .D. 



44 


10. 


.1351 


0.5054 


45 


11. 


3243 


1.8215 


46 


10. 


5811 


1.1468 


47 


. 10. 


4459 


0. 9672 


48 


10. 


5270 


1.0882 


49 


11. 


3378 


1.8820 


50 


10. 


8513 


1.8556 


51 


10. 


4459 


1.1121 


52 


11. 


,2297 


1.7324 


53 


13. 


1486 


3.2966 


54 


10. 


1216 


0.5957 


55 


12. 


9594 


3.4855 


56 


10. 


4054 


1.0326 


57 


11. 


4594 


) .7532 


58 


10. 


4459 


1.0222 


59 


10. 


3243 


0.8775 


60 


319. 


3337 


17.1500 


61 


3. 


5811 


2.7548 


62 


6. 


5676 


4.4846 


63 


5. 


5135 


4.7405 


64 


0. 


0270 


0.2325 


65 


0.0000 


0.0000 


66 


146. 


5944 


124.8155 


67 


9. 


1621 


4.3132 



Mean 


Max . 


Min . 


Rang' 


0. 0588 


13 


10 


3 


0. 2118 


17 


10 


7 


0. 1 333 


16 


9 


7 


0.1124 


14 


10 


4 


0. 1265 


14 


10 


4 


0. 2188 


17 


10 


7 


0.2157 


23 


10 


13 


0.1293 


16 


10 


6 


0.2014 


17 


10 


7 


0.3832 


25 


10 


15 


0.0692 


13 


1C 


3 


0.4052 


27 


9 


18 


0.1200 


15 


10 


5 


0.2038 


17 


10 


7 


0.1188 


15 


10 


5 


0.1020 


15 


10 


5 


1.9936 


371 


292 


79 


0.3202 


13 


0 


13 


0.5213 


21 


1 


20 


0.5511 


18 


0 


18 


0.0270 


2 


0 


2 


0.0000 


0 


0 


0 


14.5095 


820 


17 


803 


0.5014 


19 


1 


18 



r 
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f . 

CROSS VALIDATION SAMPLE (AT 1 s and BT's) 
LOWER CRITERION GROUP (N=74) 
19S- JUSTIFICATION SECTION 



No. of 



S.E. of 



iable 


Mean 


S.D. 


Mean 


Max. 


Min. 


Rangi 


1 


0.0541 


0.2815 


0.0327 


2 


0 


2 


2 


0.0135 


0.1162 


0.0135 


1 


0 


1 


3 


0.2027 


0.6406 


0.0745 


4 


0 


4 


4 


0.0405 


0.1986 


0.0231 


1 


0 


1 


5 


0.0405 


0.1986 


0.0231 


1 


0 


1 


6 


0.0270 


0.232.5 


0.0270 


2 


0 


2 


7 


0.0135 


0.1162 


0.0135 


1 


0 


1 


8 


0.0000 


0.0000 


0.0000 


0 


0 


0 


9 


0.0270 


0.1633 


0.0190 


1 


0 


1 


10 


0.1757 


0.5578 


0.0648 


3 


0 


3 


11 


0.2838 


0.6728 


0.0782 


3 


0 


3 


12 


0.1081 


0.4242 


0.0493 


3 


0 


3 


13 


0.0270 


0.1633 


0.0190 


1 


0 


1 


14 


0.0135 


0.1162 


0.0135 


1 


0 


1 


15 


0.1892 


0.6553 


0.0762 


5 


0 


5 


16 


0.0676 


0.3021 


0.0351 


2 


0 


2 


17 


' 0.1081 


0.4553 


0.0529 


3 


0 


3 


18 


0.0135 


0.1162 


0.0135 


1 


0 


1 


19 


0.1081 


0.4242 


0.0493 


2 


0 


2 


20 


0.0676 


0.3021 


0.0351 


2 


0 


2 


21 


0.0270 


0.1633 


0.0190 


1 


0 


1 


22 


0.1216 


0.3684 


0.0428 


2 


0 


2 


23 


0.0946 


. 0.3380 


0.0393 


2 


0 


2 


24 


0.0135 


0.1162 


0.0135 


1 


0 


1 


25 


0.1351 


0.4480 


0.0521 


2 


0 


2 


26 


0.0405 


0.1986 


0.0231 


1 • 


0 


1 


27 


0.0946 


0.3763 


0.0437 


2 


0 


2 


28 


0.0135 


0.1162 


0.0135 


1 


0 


1 


29 


0.0405 


0.1986 


0.0231 


l 


0 


1 


30 


2.1621 


3.5309 


0.4163 


16 


0 


16 


31 


10.1216 


0.6611 


0.0769 


15 


10 


5 


32 


10.0135 


0.1162 


0.0135 


11 


10 


1 


33 


10.3919 


1.3830 


0.1608 


20 


10 


10 


34 


10.0946 


0.5012 


0.0583 


13 


10 


3 


35 


10.0676 


0.3445 


0.0400 


12 


10 


2 


36 


10.0676 


0.5812 


0.0676 


15 


10. 


5 


37 


10.0270 


0.2325 


0.0270 


12 


10 


2 


38 


10.0000 


0.0000 


0.0000 


10 


10 


0 


39 


10.0540 


0.3661 


0.0426 


13 


10 


3 


40 


10.3513 


1.2212 


0.1420 


16 


9 


7 


41 


10.5946 


1.5432 


0.1794 


17 


9 


8 


42 


10.2297 


0.8528 


0.0991 


15 


10 


5 


43 


10.0676 


0.4165 


0.0484 


13 


10 


3 



9 
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CROSS VALIDATION SAMPLE (AT ' s and BT's) 
LOWER CRITERION GROUP (N=74) 
19S-JUSTIFICATION SECTION 
(Continued) 



No. of 

Variable Mean s.D. 

44 10.0405 0.348 

45 10.4730 1.52 .• 

46 10.1081 . 0.53o 

47 10.2162 0.9547 

48 10.0270 0.2325 

49 10.2703 1.0765 

50 10.1351 0.6265 

51 10.0540 0.3265 

52 10.2162 0.6677 

53 10.1621 0.5974 

54 9.9730 0.2325 

55 10.2297 0.7503 

56 10.0811 0.3971 

57 10.1486 0.6123 

58 10.0405 0.3487 
59... 10.0676 0.3822 
60'. 294.1848 6.8917 

61 0.6486 1.2212 

62 0.9595 1.8464 

63 0.5135 1.0500 

64 0.0270 0.1633 

65 0.0135 0.1162 

66 17.6754 27.2311 

67 1.6081 2.4372 



S.E. of 



Mean 


Max. 


Min. 


Rang* 


n n /. n c 
U . U4lo 


13 


10 


3 


n i "7 "7 a 
U. 17/0 


21 


10 


11 


U . Uo25 


14 


10 


4 


U . 111U 


17 


10 


7 


n c\ o t r\ 
U . U270 


12 


10 


2 


U . IZdI 


16 


10 


6 


n moo 


14 


10 


4 


n noon 


12 


10 


2 




Id 


10 


3 


0.0694 


13 


10 


3 


0.0270 


10 


8 


2 


0.0872 


14 


10 


4 


0.0462 


12 


10 


2 


0,0712 


14 


10 


4 


0.04 5 


13 


10 


3 


0.0444 


13 


10 


3 


0.8011 


324 


287 


37 


0.1420 


7 


0 


7 


0.2146 


10 


0 


10 


0.1221 


5. 


0 


5 


0.0190 


1 


0 


1 


0.0135 


1 


0 


1 


3.1656 


112 


0 


112 


0.2833 


9 


0 


9 



9 
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GENERALIZATION SAMPLE (CS ! s and RM's) 
UPPER CRITERION GROUP (N=74) 
19R-EVALUATION SECTION 



No. of 

Variable Mean S . D . 



1 


0.2973 


0.5163 


2 


0.3378 


0.8802 


3 


0.9054 


1.4634 


4 


0.1351 


0.3819 


5 


0.2703 


0.6884 


6 


0.0676 


0.2527 


7 


0.1081 


0.3126 


8 


0.0135 


0.1162 


9 


0.3243 


0. 7043 


10 


0.2703 


0.6682 


11 


1.0405 


1. 7629 


12 


0.2973 


\j . \j j j j 


13 


0. 0405 


0 1 986 


14 


0.0541 


o ?m s 


15 


0.1757 


0 4490 


16 


0. 2162 




17 


0 . 2027 




18 


0 .3919 




19 


0.2297 


0 Aft 

U a 40 J7 


20 


0.1351 




21 


0.0811 


o ^61 n 

V-/ a J U X W 


22 


0. 7703 


l ni 4? 


23 


1.5000 


1.5011 


24 


0.2703 


0.4768 


25 


0.9459 


1 . 2038 


26 


0.1622 


0.4388 


27 


0.3514 


0. 6288 


28 


0.2838 


0.5857 


29 


0.2162 


0.4760 


30 


10.0946 


9.6927 


31 


10.6621 


1.2306 


32 


10.6892 


1.7511 


33 


11.7973 


3.0965 


34 


10.2973 


0.9025 


35 


10.5540 


1.3461 


36 


10.1486 


0.5657 


37 


10.1892 


0.5893 


38 


10.0270 


0.2325 


39 


10.6216 


1.3619 


40 


10.5946 


1.5253 


41 


11.7703 


3.3085 


42 


10.5811 


1.2497 


43 


10.0676 


0.3822 



1 led. LI 


Max . 


Min . 


Range 


\J a UDUU 


9 
z 


n 
u 


n 

z 


o i n? ^ 

V a JL U Z. J 


4 


n 
u 


/. 
4 


n 1 7Di 


7 

1 


n 
u 


-7 

/ 


0 0444 


9 
z 


n 
u 


9 


o nsnn 


A 
4 


n 
u 


A 

4 


D D9QA 


1 


n 
u 


1 
X 


U a UJOJ 


1 


n 
u 


1 
X 


o m i s 

U a UlJJ 


1 

X 


n 
u 


1 
X 


n nfti q 




u 


4 


n ri7 7 7 

U a U / / / 


0 
J 


u 


0 
J 


D 9DAQ 

U a Z U4 -7 


1 n 
xu 


n 
U 


XU 


u a u / JO 


Q 
J 


U 


J 


u a UZ JX 


X 


U 


1 


n m 9 7 
u a u j z / 


n 
Z 


U 


z 


U a U JZZ 


z 


r\ 

0 


2 


n q & 
u . uooo 


z 


U 


z 


U . UoUo 


0 


0 


3 


n no c: q 

u . uy jy 


r 
J 


0 


5 


U . Ujdj 


z 


0 


2 




3 


0 


3 


U . U4zU 


z 


0 


2 


n i i "7 o 

u . 11/ y 


4 


0 


4 


U • 1/4 J 


/ 


U 


7 


U . Uj J4 


z 


U 


z 


n 1 ^QQ 
u . 1 jy y 


c 
J 


n 
U 


c 
J 


n nsi n 

U . UJ1U 


0 
Z 


n 
U 


0 
Z 


n n7 ^1 

U . U/ j1 


Q 
J 


U 


3 


u . uoox 


O 
J 


n 
U 


j 




9 
z 


n 
u 


9 
z 


1.1268 


63 


0 


63 


0.1431 


15 


10 


5 


0.2036 


19 


10 


9 


0.3600 


29 


10 


19 


0.1049 


15 


10 


5 


0.1565 


17 


10 


7 


0.0658 


13 


10 


3 


0.0685 


13 


10 


3 


0.0270 


12 


10 


2 


0,1583 


. 18 


10 


8 


0.1773 


18 


10 


8 


0.3846 


32 


10 


22 


0.1453 


16 


10 


6 


0.0444 


13 


10 


3 . 
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GENERALIZE SAMPLE (CS's and RM's) 

UPPER TERION GROUP (N=74) 
19K u VALUATION SECTION 
(Continued) 



No. of 






S 


.E. of 








Variable 


Mean 


S.D. 


Mean 


Max. 


Min. 


Range 


44 


10.1081 


0.5630 


0 


.0654 


' 14 


10 


4 


45 


10.3784 


1.0942 


0 


.1272 


16 . 


10 


6 


46 


10.3649 


0.9872 


0 


.1148 


15 


10 


5 


47 


10.3513 


0.8980 


0 


.1044 


14 


10 . 


4 


48 


10.8378 


1.6719 


0 


.1944 


19 


10 \ 


9 


49 


10.4594 


1.1003 


0 


.1279 


16 


10 


6 


50 


10.2162 


0.7811 


0 


.0908 


15 


10 


5 


51 


10.1081 


0.5120 


0, 


.0595 


13 


10 


3 


52 


11.5946 


2.0668 


0, 


.2403 


18 


10 


8 


53 


13.5405 


3.7968 


0, 


.4414 


31 


10 


21 


54 


10.8108 


1.4304 


0, 


.1663 


16 


10 


6 


55 


11.6892 


2.2141 


0, 


.2574 


20 


10 


10 


56 


10.3108 


0.9641 


0. 


.1121 


16 


10 


6 


57 


10.6757 


1.2939 


0. 


.1504 


17 


10 


7 


58 


10.5000 


1.1849 


0. 


.1377 


17 


10 


7 


59 


10.4865 


1.1133 


0. 


.1294 


15 


10 


5 


60 


310.5364 


21.1002 


2. 


.4528 


445 


290 


155 


61 


2.9730 


4.0102 


0. 


,4662 


32 


0 


32 


62 


4.4189 


4.2716 


0. 


.4966 


19 


0 


19 


63 


2.6892 


3.4240 


0. 


3980 


16 


0 


16 


64 


0.0135 


0.1162 


0. 


0135 


1 


0 


1 


65 


0.0000 


0.0000 


0. 


,00 0 0 


o. 


0 


0 


66 


98.1754 


83.3072 


9. 


6843 


530 


0 


530 


67 


6.3378 


4.7637 


0. 


5538 


22 


0 


22 
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GENERALIZATION SAMPLE (CS's and RM ? s) 
MIDDLE CRITERION GROUP (N=74) 
19R-EVALUATION SECTION 



No. of 

Variable Mean S.D. 



1 

X 


n 


A tr Qtr 


n 

U . 


AftA9 


9 
z 


n 

U i 


• J JlH 


0 

vj , 


801 9 

• O U JLZ 


J 


n 


ft aaq 


1 

JL . 


1 1ft7 
, X JO / 


4 


n 


no/, A 
• uy40 


n 

u . 


9QA7 

iZ74/ 


J 


n 


1 699 
• XOZZ 


0 

VJ , 


AOAA 


A 
0 


0 


1 /i ft A 
. X400 


U , 


R1 RO 
. JlJU 


7 


u . 


9 09 7 
. Z UZ / 


u . 


R9 9 Q 

, JZZ7 


Q 

o 


u , 


. U4UD 


u , 


1 QftA 


Q 


u . 


9 9 Q7 

, zzy / 


n 


. 434 / 


i o 


u . 


1 A99 
. XOZZ 


n 


A Q7 T 

,47 / J 


XX 


n 


A 1 R1 


x . 


9 009 
, ZUUZ 


1 9 

xz 


n 


909 7 
. zuz / 


n 


A A7 A 
.40/0 


i ^ 

ij 


n 


ni 7^ 

. UJL JD 


n 


1 1 A9 
, 11DZ 


X4 


n 


ORA1 
. U34X 


n 


9 9 77 
, ZZ / / 


1 s 


n 
vj . 


1 91 A 
• XZ 10 


n 
u . 


1 AftA 
, J 004 


1 A 
XO 


n 


1 ftQ9 

. lO -7Z 


n 

u . 


QQ/, O 

, j?4 J 


1 7 


n 

u . 


OQAA 

. uy40 


0 
u . 


77 ft O 


1 ft 


n 

u . 


9ft ^ft 
. ZO JO 


n 
u . 




1 Q 


n 

u . 


A1 ftQ 


n 
u . 


7 9 1 A 

, / Z ID 


zo 


n 

u . 


1 91 A 
. XZXO 


n 


7 Aft A 
, J004 


9 1 

Z JL 


n 

u . 


9 09 7 
. ZUZ / 


n 

u . 


A A7 A 
,40/0 


9 9 

ZZ 


n 

u . 


AftQ9 


1. 


,1339 


9 ^ 
Z j 


1 

X • 


19 Al 
. jZ4j 


1. 


,4440 


9A 
Z4 


n 

u . 


1 A99 

. JL OZZ 


0. 


,4064 


9 S 

Z _) 


n 
u • 


QftAR 

■ 700J 


i. 


,2219 


26 


0. 


0541 


0. 


2277 


27 


0. 


1622 


0. 


4064 


28 


0. 


1757 


0. 


4490 


29 


0. 


1216 


0. 


3291 


30 


8. 


,7297 


5. 


5967 


31 


11. 


0135 


1. 


4571 


32 


10. 


5811 


1. 


5439 


33 


11. 


5946 


2. 


1891 


34 


10, 


2162 


0. 


7075 


35 


10. 


,2838 


0. 


7498 


.36 


10. 


,2973 


0. 


9613 


37 


10. 


,3513 


V 

1_ . 


,0782 


38 


10. 


,0811 


0. 


,3971 


39 


10. 


.3648 


0. 


.8205 


40 


10. 


.2973 


1.0301 


41 


10. 


.8649 


1, 


,5644 


42 


10. 


,2703 


0. 


.6682 


43 


10. 


.0270 


0. 


,2325 



S.E. of 



Mean 


Max ■ 


Min ■ 


R an r>, 
i\ d 1 1 ^ i 


u ■ / y o 


-i 


o 


■J 


VJ • 7 J JL 


A 


O 

VJ 


A 

4 


0 1 ^9A 

■ JL J Z *r 




o 

VJ 


-/ 


0 01A1 
U • U J 4 J 


1 

JL 


O 

VJ 


1 

X 


0 OA 7 9 

VJ a VJH / Z. 


9 

£. 


o 
u 


2 


0 ORQQ 


-\ 
J 


o 

VJ 


J 


0 OAOft 
U ■ UO UO 


9 
z 


o 

u 


9 
z 


O 09 11 
U ■ UZ jl 


i 

X 


n 

u 


X 


O 0R9 Q 

u . u_>z y 


9 


o 

u 


9 
Z 


O OR7ft 
U ■ U_) / O 


J 


0 
u 


J 


O 1 7Q R 
U ■ JL J y D 


c: 
J 


o 
u 


c: 
3 


O OR AA 
U ■ U_)44 


9 
z 


o 

U 


9 
Z 


0 01 1 R 
U . UlJJ 


1 

X 


o 

u 


X 


0 09 A S 
U ■ UZ O J 


1 

X 


o 

u 


1 

X 


0 OA 9 ft 

U ■ U4ZO 


9 

z 


n 

VJ 


9 
z 


0 OASft 

U ■ U4-JO 


1 

X 




X 


O 07Q 7 


9 
z 


o 
u 


9 
• Z 


O O A 9 A 
U . U0Z4 


9 
Z 


o 
u 


Z 


O Oft TO 

u . uo jy 


Q 

J 


o 
u 




O OA 9ft 
U . U4Z0 


9 
Z 


o 
u 


Z 


O ORA A 
U ■ U_)44 


9 
Z 


o 
u 


9 
Z 


O 1 11 ft 
U . IjIO 


A 
O 


o 
u 


z: 
0 


O 1 A7 Q 
U. ID/ 7 


o 
O 


o 
u 


* ft 
O 


0 OA7 9 
U . U4 / Z 


9 
z 


o 
u 


9 
Z 


0 1 A90 

VJ ■ JL4 Z U 




n 

u 


J 


U ■ UZ O J 


1 

JL 


o 

VJ 


1 

X 


O OA7 9 

U • U4 / Z 


9 
z 


o 
u 


9 
z 


0.0522 


2 


0 


2 


0.0383 


1 


0 


1 


0.6506 


26 


0 


26 


.0.1694 


15 


10 


5 


0.1795 


•18 


7 


11 


0.2545 


19 


10 


•9 


0.0822 


_3 


10 


3 


0,0872 


13 


10 


3 


0.1117 


14 


10 


4 


0.1253 


16 


9 


7 


0.0462 


12 


10 


2 


0.0954 


13 


10 




0.1197 


15 


9 


6 


0.1819 


16 


10 


6 


0.0777 


13 


10 


3 


0.0270 


12 


10 


• 2 
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GENERALIZATION SAMPLE (CS 1 s and RM f s) 
MIDDLE CRITERION GROUP (N=74) 
1 9 R- E VALU AT 10 N SECTION 
(Continued) 



No. of 
Variable Mean 



44 


10, 


.1081 


45 


10, 


.2838 


46 


10, 


.3513 


47 


10, 


.1351 


48 


10, 


.4865 


49 


10, 


.6216 


50 


10, 


.2027 


51 


10, 


.2567 


52 


11, 


.4324 


53 


12, 


.7162 


54 


10, 


.4865 


55 


11, 


.7432 


56 


10. 


.0811 


57 


10. 


,1892 


58 


10. 


.2567 


59 


10. 


.2567 


60 


305. 


,8467 


61 


1. 


,8513 


62 


3. 


8784 


63 


2. 


,7703 


64 


0. 


,2297 


65 


0. 


,0000 


66 


84, 


.5132 


67 


r 


.8648 



S.E. of 
S.D. Mean 



0.4553 0.0529 

0.8524 0.0991 

0.7840 0.091.1 

0.4776 0.0555 

0.9826 0.1142 

1.1786 0.1370 

0.6617 0.0769 

0.6424 -0.0747 

2.3059 0.2681 

2.9633 0.3445 

1.2191 0.1417 

2.2638 0.2632 

0.3971 0.0462 

0.5893 0.0685 

0.6837 0.0795 

0.7034 0.0818 

10.0993 1.1740 

1.7570 0.2043 

3.0608 0.3558 

2.5831 0.3003 

0.8687 0.1010 

0.0000 . 0.0000 

53.0552" 6.1675 

3.3365 0.3879 



Max. Min. Range 



12 


10 


2 


14 


10 


4 


13 


10 


3 


12 


10 


2 


14 


10 


4 


15 


9 


6 


14 


10 


4 


13 


10 


3 


22 


10 


12 


26 


10 


16 


16 


10 


6 


20 


10 


10 


13 


10 


3 


13 


9 


4 


13 


10 


3 


13 


10 


3 


338 


290 


48 


7 


0 


7 


11 


0 


11 


11 


0 


11 


6 


0 


6 


0 


0 


0 


293 


0 


293 


14 


0 


14 
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GENERALIZATION SAMPLE (CS's and RM's) 
LOWER CRITERION GROUP (N=74) 
19R- EVALUATION SECTION 



No. of S.E. of 

Variable Mean S.D. Mean Max. Min. Range 



1 


0. 


,2297 


0. 


4547 


0. 


,0529 


2 


0 


2 


2 


0. 


.3649 


0. 


,6532 


0. 


.0759 


3 


0 


3 


3 


0. 


.8378 


1. 


1229 


0. 


.1305 


6 


0 


6 


4 


0. 


,1486 


0. 


,4279 


0. 


.0497 


2 


0 


2 


5 


0. 


.2432 


0. 


,4320 


0. 


.0502 


1 


0 


1 


6 


0. 


.0270 


0. 


,1633 


0. 


.0190 


1 


0 


1 


7 


• 0. 


.1622 


0. 


,4388 


0. 


.0510 


2 


0 


2 


8 


0. 


,0135 


0. 


,1162 


0. 


,0135 


1 


0 


1 


9 


0. 


.2703 


0. 


,5311 


0. 


.0617 


2 


0 


2 


10 


0. 


,4324 


0. 


,7038 


0. 


.0818 


3 


0 


3 


11 


0. 


.8378 


1. 


1229 


0. 


,1305 


5 


0 


5 


12 


0. 


.1892 


0. 


4277 


0. 


,0497 


2 


0 


2 


13 


0. 


,0000 


0. 


00 0 0 


0. 


.0000 


0 


0 


0 


14 


0. 


,1216 


0. 


,4038 


0. 


.0469 


2 


0 


2 


15 x 


0. 


.1757 


0. 


,4785 


0. 


.0556 


3 


0 


3 


16 


0. 


,1351 


0. 


,3819 


0. 


,0444 


2 


0 


2 


17 


0. 


,0946 


0. 


3380 


0. 


,0393 


2 


0 


2 


13 


0. 


,2162 


0. 


4463 


0. 


,0519 


2 


0 


2 


19 


0. 


,4189 


0. 


7586 


0. 


,0882- 


3 


0 


3 


20 


0. 


,1757 


0. 


4785 


0. 


,0556 


2 


0 


2 


21 


0. 


,1351 


0. 


3442 


0. 


,0400 


1 


0 


1 


22 


0. 


,6486 


0. 


9280 


" 0. 


,1079 


4 


0 


4 


23 


1. 


,6621 


1. 


5374 


0. 


,1787 


7 


0 


7 


24 


0. 


.1081 


0. 


3537 


0. 


.0411 


2 


0 


2 


25 


0. 


9865 


1. 


1878 


0. 


,1381 


4 


0 


4 


26 


0. 


,1216 


0. 


3684 


0. 


,0428 


2 


0 


2 


27 


0. 


.3784 


0. 


6127 


0. 


,0712 


3 


0 


3 


28 


0. 


,1622 


0. 


4388 


0. 


,0510 


? 


0 


2 


29 


0.1351 


0. 


3442 


0. 


0400 


1 


0 


1 


30 


9. 


.4324 


5. 


0965 


0. 


,5925 


29 


0 


29 


31 


10. 


.4189 


0. 


,9364 


0.1089 


13 


9 


4 


32 


10. 


.5676 


1. 


0479 


0. 


.1218 


14 


10 


4 


33 


11. 


0946 


2. 


1592 


0. 


2510 


23 " 


7 


16 


34 


10.2838 


0.8196 


0. 


0953 


14 


10 


4 


35 


10. 


,4730 


0.8793 


0. 


1022 


13 


10 


3 


36 


10. 


,0676 


0. 


4165 


0. 


0484 


13 


10 


3 


37 


10. 


.2838 


0.8027 


0. 


,093? 


14 


10 


4 


38 


10. 


.0270 


0. 


,2325 


0. 


,0270 


12 


10 


2 


39 


10, 


.3919 


0.9189 


0.1068 


14 


9 


5 


40 


10. 


.6081 


1. 


,1915 


0. 


.1385 


15 


9 


6 


41 


11.1486 


2. 


.0385 


0. 


.2370 


19 


8 


11 


42 


10.3108 


0. 


8264 


0. 


0961 


13 


9 


4 


43 


10. 


.0000 


0. 


oooo 


0. 


,0000 


10 


10 


0 
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GENERALIZATION SAMPLE (CS T s and RM T s) 
LOWER CRITERION GROUP (N=74) 
19R- EVALUATION SECTION 
(Continued) 



No. of 
Variable Mean 



44 


10.0811 


45 


10.3784 


46 


10.1757 


47 


10.1486 


48 


10.3378 


49 


10.7432 


50 


10.2703 


51 


10.17 r " 


52 


11.1351 


53 


13.1757 


54 


10.2567 


55 


11.3378 


56 


10.2027 


57 


10.7567 


58 - 


10.2162 


59 


10.2432 


60 


305.^/64 


61 


1.5540 


62 


4.2838 


63 


2.8513 


64 


0.7297 


65 


0.0135 


66 


100.2835 


67 


6.3648 



S.E. of 
S.D. Mean 



0.5675 0.0660 

1.0689 0.12.43 

0.6487 0.0754 

0.5410 0.0629 

0.7635 0.0888 

1.4340 0.1667 

0.7816 0.0909 

0.4785 0.0556 

1.7698 0.2057 

3.3407 0.3884 

1.0476 0.1218 

2.0155 0.2343 

0.6617 0.0769 

1.4026 0.1631 

0.6677 0.0776 

0.6582 0.0765 

9,0569 1.0528 

1.5362 0.1786 

2.6767 0.3112 

2.4700 0.2871 

1.5641 0.1818 

0.1162 0.0135 

54.5660 6.3432 

2.8020 0.3257 



Max. Min . Range 



13 


8 


5 


17 


10 


7 


14 


9 


5 


13 


10 - 


3 


13 


10 


3 


17 


10 


7 


14 


10 


4 


12 


10 


2 


18 


9 


9 


22 


8 


14 


16 


8 


8 


18 


8 


10 


13 


10 


3 


17 


9 


8 


14 


10 


4 


13 


10 


3 


333 


290 


43 


7 


0 


7 


16 


0 


16 


9 


0 


9 


9 


0 


9 


1 


0 


1 


353 


0 


353 


14 


0 


14 
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GENERALIZATION SAMPLE (CS's and RM ? s) 
UPPER CRITERION GROUP (N=74) 
19S- JUSTIFICATION SECTION 



No. of 

Variable Mean S*D. 



1 


0. 


.5135 


0, 


.8951 


2 


0. 


.8243 


1, 


.3381 


3 


2, 


.0811 


1, 


.9846 


4 


0, 


.2568 


0. 


,6633 


5 


0. 


.6081 


1. 


,0831 


6 


0, 


.1622 


0, 


.5241 


7 


0, 


.1757 


0. 


.3831 


8 


0. 


.0946 


0. 


.2947 


9 


0. 


.9189 


1. 


,1196 


10 


1. 


.2567 


1, 


.8061 


11 


2. 


,4865 


2. 


,7411 


12 


0, 


.8784 


0, 


.9209 


13 


0, 


.1*86 


0, 


.3946 


14 


0, 


.0811 


0, 


.2748 


15 


0. 


,8784 


0, 


.8101 


16 


0. 


,8649 


0, 


.880': 


17 


0. 


,3649 


0. 


.6739 


18 


0. 


,5946 


0. 


,8263 


19 


1. 


,4189 


1. 


,3650 


20 


0. 


,6351 


0. 


,8848 


21 


0, 


.2838 


0. 


,5369 


22 


1, 


.4054 


1, 


,5955 


23 


2, 


.5270 


2. 


,0555 


24 


0, 


. 0135 


. 0. 


,1162 


25 


2, 


.6757 


2. 


,4220 


26 


0. 


.4595 


0, 


.6660 


27 


1. 


,1351 


1. 


,1387 


28 


0. 


,2973 


0. 


,5669 


29 


0. 


,5135 


0. 


,7070 


30 


24. 


,5673 


15. 


38-': 5 


31 


11. 


,2027 


2. 


,0804 


32 


11. 


,4189 


2. 


,3815 


33 


14. 


,0540 


4, 


,0776 


34 


10, 


,4594 


.1. 


,1958 


35 


11, 


.0811 




.8635 


36 


10, 


.3378 


1. 


,0634 


37 


10, 


.3108 


0, 


.7389 


38 


10, 


.i621 


0, 


.5496 


39 


11, 


.7567 


. 2, 


.2071 


40 


12, 


.5811 


3. 


,5654 


41 


14, 


,*621 


4, 


.2500 


42 


11, 


.7027 


1. 


,9705 


43 


10, 


.2703 


0.7457 



S.E. of 



Mean 


Max. 


Min . 


Rang- 

.. ... .J3L 


0.1041 


4 


0 


4 


0.1556 


S 


0 


8 


0.2307 


9 


0 


9 


0.0771 


3 


0 


° 3 


0.1259 


6 


0 


f. 


0.0609 


3 


0 


3 


0.0445 


1 


0 


1 


0.0343 


1 


0 


1 


0.1302 


5 


0 


5 


0.2100 


9 


0 


9 


0.3186 


14 


0 


14 


0.1070 


4 


0 


4 


0. 0459 


2 


0 


2 


0.0319 


1 


0 


1 


0.0942 


4 


0 


4 


0.1024 


4 


0 




0. 0783 


2 


0 


2 


0.0961 


3 


0 


3 


0.1587 


6 


0 


6 


0.1029 


4 


0 


4 


0.0624 


2 


0 


2 


0. 1855 


6 


o 


6 


0. 2390 


8 


0 


8 


0.0135 


1 


o 


1 


0.2815 


10 


0 


10 


0. 0774 


3 


o 


3 


0.1324 


4 


0 


' L 


0.0659 


2 


0 


2 


0.0822 


3 


0 


3 


1.7884 


75 


2 


73 


0.2418 


20 


10 


10 


0.2768 


24 


10 


14 


0.4740 


27 


10 


17 


0.1390 


16 


10 


6 


0.2166 


20 


10 


10 


0.1236 


16 


10 


6 


0.0859 


13 


10 


3 


0.0639 


13 


10 


3 


0.2566 


20 


10 


10 


0.4345 


27 


9 


18 


0.4 >40 


29 


10 


19 


0.2291 


- U 


10 


10 


0.0867 


13 


10 


3 



0 
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GENERALIZATION SAMPLE (CS's and RM's) 
UPPER CRITERION GROUP (N=74) 
19S- JUSTIFICATION SECTION 
(Continued) 



No. of S.E. of 



riable 


Mean 


S 


.D. 


Mean 


Max . 


Min. 


Range 


44 


10 


.1621 


0 


.5974 


0, 


.0694. 


13 


10 


3 


45 


11 


.9594 


1 


.7629 


0, 


.2049 


17 


10 


7 


46 


11 


.3648 


1, 


.4858 


0, 


.1727 


16 


10 


6 


47 


10 


.5811 


1, 


.1104 


0, 


.1291 


14 


10 


4 


48 


11, 


.0946 


1, 


.6316 


0, 


.1897 


16 


10 


6 


49 


12, 


.6081 


2, 


.6266 


0, 


.3053 


22 


10 


12 


50 


11, 


.1757 


1, 


.6163 


0, 


.1879 


16 


10 


6 


51 


10. 


.3784 


0, 


.7711 


0, 


.0896 


13 


10 


3 


52 


12, 


.7838 


3. 


,3445 


0. 


,3888 


23 


10 


13 


53 


15. 


,8647 ■ 


4, 


.6855 


0. 


,5447 


30 


10 


20 


54 


10. 


,0405 


0, 


.3487 


0. 


,0405 


13 


10 


3 


55 


14. 


,5811 


*> 

J . 


,8924 


0. 


,4525 


23 


10 


13 


56 


10. 


,7432 


1. 


,2612 


0. 


,1466 


16 


10 


6 


57 


12, 


.2838 


2. 


,3671 


0. 


,2752 


18 


10 


8 


58 


10. 


,5000 


1, 


.0102 


0. 


,1174 


14 


10 


4 


59 


11. 


,0135 


1. 


,4665 


0. 


,1705 


15 


10 


5 


60 


337. 


,1443 


28. 


,4735 


3. 


,3100 


423 


294 


129 


61 


6. 


,0946 


4. 


,5183 


0. 


,5252 


20 


0 


20 


62 


10. 


,5811 


6. 


,9420 


0. 


,8070 


32 


1 


31 


63 


7. 


,8243 


6. 


,8012 


0. 


,7906 


31 


0 


31 


64 


0, 


.0676 


0. 


,4778 


0. 


,0555 


4 


0 


4 


65 


0, 


.0000 


0. 


,0000 


0. 


,0000 


0 


0 


0 


66 


228. 


,2968 


155- 


,7997 


18. 


1113 


881 


17 


864 


67 


12. 


,4730 


4. 


,6852 


0. 


,5446 


n o 


1 


21 
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GENERALIZATION SAMPLE (CS ! s and RM f s) 
MIDDLE CRITERION GROUP (N=74) 
19S- JUSTIFICATION SECTION 



No. of S.E. of 



riable 


Mean 


S.D. 


Mean 


Max. 


Min. 


Rang* 


1 


0.3108 


0.639: 


0.0743 


2 


0 


2 


2 


0.3'§r9 


0.8246 


0.0959 


3 


0 


3 


3 


1.0946 


1.6232 


0.18S7 


7 


0 


7 


4 


0.2027 


0.6617 


0.0769 


u 


0 


4 


5 


0.4324 


0.7417 


0.0862 


3 


0 


3 


6 


0.1081 


0.5381 


0.0625 


4 


0 


4 


7 


0.1622 


0.4690 


0.0545 


3 


0 


3 


8 


0.0405 


0.2585 


0.0301 


2 


0 


2 


9 


0.4865 


0.7806 


0 0907 


3 


0 


~" 3 


10 


0.4324 


0.8289 


0.0964 


4 


0 


4 


11 


1.5676 


1.7910 


0.2082 


7 


0 


7 


12 


0.4054 


0.7199 


0.0837 


3 


0 


3 


13 


0.1892 


0.5893 


0.0685 


3 


0 


3 


14 


0.0811 


0.3208 


0.0373 


2 


0 


2 


15 


0.7973 


0.9506 


0.1105 


4 


0 


4 


16 


0.4730 


0.6458 


0.0751 


3 


0 


3 


17 


0.1892 


0.4875 


0.0567 


3 


0 


3 


18 


0 2297 


0.5376 


0.0625 


3 


0 


3 


19 


0.3919 


0.9871 


0.1148 


4 


0 


4 


20 


0.3649 


0.7508 


0.0873 


3 


0 


3 


21 


0.2703 


0.5563 


0.0647 


2 


0 


2 


22 


0.5811 


0.9509 


0.1105 


5 


0 


5 


23 


2.0270 


1.8870 


0.2194 


11 


0 


11 


24 


0.0000 


0.0000 


0.0000 


0 


0 


0 


25 


1.4324 


1.6393 


0.1906 


7 


0 


7 


26 


0.2162 


0.5304 


0.0617 


3 


0 


3 


27 


0.5405 . 


0.7251 


0.0843 


3 


0 


3 


28- 


0.1622 


0.3711 


0.0431 


1 


0 


1 


29 


0.2.838 


0.5369 


0.0624 


3 


0 


3 


30 


14.3648 


9.8961 


1.1504 


49 


0 


49 


31 


10.7567 


1.5860 


0.1844 


16 


10 


'6 


32 


10.6621 


1.4456 


0.1680 


16 


9 


7 


33 


12.1351 


3.2449 


0.3772 


27 


10 


17 


34 


10.3243 


1.0351 


0.1203 


16 


10 


6 


35 


10.8108 


1.5141 


0.1760 


16 


10 


6 


36 


10.1892 


0.9017 


0.1048 


16 


10 


6 


37 


10.2838 


0.9583 


0.1114 


17 


10 


7 


38 


10.0541 


0.3265 


0.0380 


12 




2 


39 


10.8513 


1.4304 


0.1663 


15 


r '*uo 


5 


40 


10.8513 


1.6021 


0.1862 


16 


10 




41 


12.7567 


3.0780 


0.3578 


21 


10 


li 


42 


10.7973 


1.5168 


0.1763 


16 


10 


6 


43 


10.2973 


0.9754 


0.1134 


16 


10 


6 
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GENERALIZATION SAMPLE (CS's and KM's) 
MIDDLE CRITERION GROUP (N=74) 
19S- JUSTIFICATION SECTION 
(Continued) 



No. of 
Variable Mean 



44 


10.1351 


45 


11.8108 


46 


10.6892 


47 


10.3513 


48 


10.3919 


49 


11.5000 


50 


10.5135 


51 


10.3784 


52 


11.0811 


53 


14.2297 


54 


10.0000 


55 


12.4324 


56 


10.3649 


57 


11.0540 


58 


10.2567 


59 


10.5676 


60 


316.5225 


61 


3.1216 


62 


6.0000 


63 


5.2027 


64 


0.0405 


65 


0.0000 


66 


133.0268 


67 


8.4730 



S.E. of 
S . D . Mean 



0.5318 0.0618 

2.3035 0.2678 

0.9782 0.1137 

0.9131 0.1061 

1.0313 0.1199 

1.8672 0.2171 

1.1010 0.1280 

0.8392 0.0975 

1.7102 0.1988 

4.1266 0.4797 

0.0000 0.0000 

2.9381 0.3415 

0.9447 0.1098 

1.6124 0.1874 

0.6424 0.0747 
1.1236 . 0.1306 

18.2112 2.1170 

2.7843 0.3237 

4.7571 0.5530 

4.5510 0.5290 

0.1986 0.0231 

0.0000 0.0000 

96.9929 11.2752 

4.2819 0.4978 



Max. Min. Range 



13 


10 


3 


21 


10 


11 


14 


10 


4 


15 


10 


5 


16 


10 


6 


19 


10 


9 


15 


10 


5 


14 


10 


4 


19 


10 


9 


33 


10 


23 


10 


10 


0 


23 


10 


13 


15 


10 


5 


16 


10 


6 


13 


10 


3 


16 


10 


6 


374 


290 


84 


14 


0 


14 


25 


0 


25 


1? 


0 


19 


1 


0 


1 


0 


O 


. 0 


442 


0 


442 


18 


0 


18 
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GENERALIZATION SAMPLE (CS 1 s and RM's) 
LOWER CRITERION GROUP (N=74) 
19 S- JUSTIFICATION SECTION 



No. of 

Variable Mean S.D. 



1 


0.0135 


0.1162 


2 


0.0676 


0.3021 


3 


0.1351 


0.5318 


4 


0.0405 


0.1986 


5 


0.0541 


0.2815 


6 


0.0135 


0.1.162 


7 


0.0676 


0.3445 


8 


0.0270 


0.1633 


9 


0.0405 


0.2585 


10 


0.2432 


0.6985 


11 


0.3514 


0.8182 


12 


0.2297 


0.6092 


12 


0.0135 


0.1162 


14 


0.0270 


0.1633 


15 


0.3243 


0.9524 


16 


0.0811 


J. 4898 


17 


.0.0541 


0.2815 


18 


0.0135 


0.1162 


19 


0.1216 


0.4038 


20 


0.0541 


0.2277 


21 


0.0676 


0.2527 


22 


0.1216 


0.4668 


23 


0.1757 


0.5819 


24 


0.0000 


0.0000 


25 


0.1216 


0.4668 


26 


0.0541 


0.2277 


27 


0.2027 


0.5729 


28 


0.013:, . 


0.1162 


29 


0.0405 


0.1986 


30 


2.7703 


4.0798 


31 


10.0270 


0.2325 


32 


10.1081 


0.4845 


33 


10.2567 


1.0' 76 


34 


10.0405 


0.1986 


35 


10.0946 


0.4731 


36 


10.0270 


0.2325 


37 


10.1351 


0.6688 


38 


10.0540 


0.3.661 


39 


10.0676 


0.4165 


40 


10.4189 


1.4046 


41 


10.8513 


1.9492 


42 


10.5135 


1.4069 


43 


10.0135 


0.1162 



Mean 


Max . 


Min. 


Rang* 


0.0135 


1 


0 


1 


0.0351 


2 


0 


2 


0.0618 


4 


0 


4 


0.0231 


1 


0 ' 


1 


0.0327 


2 


0 


2 


0.0135 


1 


0 


1 


0. 0400 


2 


0 


2 


0.0190 


1 


0 


1 


0.0301 


2 


0 


2 


0,0812 


4 


0 


4 


0.0951 


4 


0 


4 


0. 0708 


3 




3 


0.0135 


1 


0 


1 


0.0190 


1 


0 


1 


0.1107 


5 


0 


5 


0 . 0' 69 


4 


o 


4 


0. 0327 


2 


o 


2 


0. 0135 


1 


o 


1 


0.0469 


2 


o 


2 


0. 0265 


1 


o 


1 


0.0294 


1 


0 


1 


0 0S4^ 


-x 
~j 


n 


-} 
-j 


0. 0676 


L 


n 


4 


0 0000 


n 

W 


n 

\j 


n 






n 




0. 0265 


1 

JL 


n 


1 


\J . uuuu 


9 
c 


n 


9 


0.0135 


1 


0 


l 


0.0231 


1 


0 


l 


0.4743 


21 


0 


21 


0.0270 


12 


10 


2 


0.0563 


13 


10 


3 


0.1218 


18 


10 


8 


0.0231 


11 


10 


1 


0.0550 


13 


10 


3 


0.0270 


12 


10 


2 


0.0777 


14 


10 


4 


0.0426 


13 


10 


3 


0.0484 


13 


10 


3 


0.1633 


18 


9 


9 


0.2266 


19 


10 


9 


0.1635 


17 


id 


7 


0.0135 


11 


10 


1 
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GENERALIZATION SAMPLE (CS f s and RM's) 
LOWER CRITERION GROUP (N=74) 
1 9 S- JUSTIFICATION SECTION 
(Continued). 



No. of 
Variable Mean 



44 


ID , 


. 0405 




10 . 


£ n n o 

. 689Z 


46 


10 . 


. 1351 


A "7 
4 / 


10 . 


. 121b 


48 


10 , 


. 027C 


49 


10. 


.2162 


50 


10. 


.1351 


• i 


10. 


.0946 


52 


10. 


.1892 


53 


10. 


2703 


54 


10. 


,0000 


55 


10. 


,2567 


56 


10. 


0676 


57 


10. 


.4054 


58 


10. 


,0270 


59 


10. 


,0o'/^ 


60 


295. 


,3472 


61 


0. 


,6216 


62 


1. 


,4189 


63 


0. 


,6892 


64 


o. 


.0405 


65 


0, 


0000 


66 


27. 


,4051 


67 


1 . 


,8648 



S.E. of 
S . D . Mean 



0.2585 0.0301 

2.1062 0.2448 

0.8492 0.0987 

0.6183 0.0719 

0.2325 0.0270 

0.7266 0.0845 

0.5811 0.0o75 

0.3763 0.0437 

0.7706 0.0896 

0.92o0 0.1076 

0.0000 0.0000 

0.9225 0.1072 

0.3021 0.0351 

1.1811 0.1373 

0.2325 0.0270 

0.3445 0.0400 

7.4695 0.8683 

0.9751 0.1133 

2.2514 0.2617 

1.5432 0.1794 

0.1986 0.0231 

0.0000 0.0000 

40.0231 4.6526 

2.5660 0.2983 



Max. Min. Range 



Iz 


ID 


I 




9 


1 


1 / 


10 


1 

I 


1 A 
14 


10 


4 


12 


10 


2 


14 


10 


4 


13 


10 


3 


12 


10 


2 


15 


10 


5 


16 


10 . 


6 


10 


10 


0 


15 


10 


5 


12 


10 


2 


15 


10 


5 


12 


10 


2 


12 


10 


2 


323 


289 


34 


4 


0 


4 


12 


0 


12 


9 


0 


9 


1 


0 


1 


0 


0 


0 


172 


0 


172 


10 


0 


10 



APPENDIX C 
CORRELATIONS AMONG THE 67 VARIABLES* 



CROSS VALIDATION SAMPLE - AT's AND BT T s COMBINED (N=222) 



EVALUATION SECTION (19R) 178 

JUSTIFICATION SECTION (19C) 184 

EVALUATION SECTION (19R) VS. 190 
JUSTIFICATION SECTION (19S) 

GENERALIZATION SAMPLE - CS's AND RM T s COMBINED (N=222) 

EVALUATION SECTION (19R) 199 

JUSTIFICATION SECTION (19S) 205 

EVALUATION SECTION (19R) VS. 211 
JUSTIFICATION SECTION (19S) 



Table 8 (pages 20-21) contains a definition for each of 
the 67 variables. Variables }. through 30 are the same as 
Variables 31 through 60 except for the method of weighting 
that was used. Therefore, these two sets of variables are 
highly correlated within the evaluation section and the 
justification section of both samples. These 30 correla- 
tion coefficients are shown in italic type in each correr 
lation matrix for the evaluation section and the justifi- 
cation section. 
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APPENDIX D 

MANN-WHITNEY U TESTS WITH ASSOCIATED z VALUES* AND 
t TESTS OP MEAN DIFFERENCE ON ALL 67 VARIABLES 



CROSS VALIDATION SAMPLE 

EVALUATION SECTION 

LOWER VS. UPPER 222 

LOWER VS. MIDDLE 225 

MIDDLE VS. UPPER 228 

JUSTIFICATION SECTION 

LOWER VS. UPPER 231 

LOWER VS. MIDDLE 234 

MIDDLE VS. UPPER 237 

GENERALIZATION SAMPLE 

EVALUATION SECTION 

LOWER VS. UPPER 240 

LOWER VS. MIDDLE 243 

MIDDLE VS. UPPER 246 

JUSTIFICATION SECTION 

LOWER VS. UPPER 249 

LOWER VS. MIDDLE 252 

MIDDLE VS. UPPER 255 



The convention that was followed in the Mann-Whitney U tests was to enter 
data into the computer program first for the lower criterion group in the 
comparison. For those z values that achieved statistical significance, the 
direction of the difference is that the higher criterion group evidenced a 
greater frequency or greater weighted frequency of the variable. 

O 

ERIC 



222 



a 

O 

w co 



w 








O 




w 


CO 




p^ 


H 




p^ 






H 


< 


.>— > 


P 




Pi 




s 


cr> 


£n 


o 


H 




a 




w 


CO 


2! 




Hi 


O 


Ph 


P 


H 


o 




H 






a 


CO 




w 


H 




CO 


en 






w 


o 




H 


H 


b 




Pfij 


H 




w 


H 




H 


< 


P 


H 


P 




Pi 






U 






Pi 


> 


00 


W 


w 


w 




p 


Ph 


W 




Ph 






p 


H 














N 


CO 


o 




> 




Q 




w 


W 






H 




Ph 


< 






H 


o 




a 




CO 


o 






CO 


w 




CO 




o 


< 




H 






H 


EC 




< 


IT 


FO 


ID 










CO 




CO 


W 


> 


H 


hh* 




CO 




CO 


W 




CO 


H 


H 








s 


P 




o 


{H 


> 


W 


W 






25 


vo 


H 


H 






H 


HH* 






AL 


& 


1 








O 





u 

•rl 

i— I i— I 

•h a) 
a) 

rQ HH* 

o 
n 

Ph 



ooooooooooooooooo 



LO LO «-0 LO LO 

o o o o o 



AAAAAAAAAAII AAAAAAVA 



A A A V A 



IT) IT) IT) l-O LO ■ l -0 C 

o o o o o o *h 



o 



pHO^pHpHpHpHpHpHpHpHpHpHp^pHpHpHpHrHpHpHpHpHp^ 

O O 
O O 



0) 
I— I 

> o 



oooc^JlnlnvDOcolncnoovDroonooa^c^J^l^|vD^HO^l^|a^^^vDH 



HO H H 



OOOrHOrHrHOOrH,-HOC0rHO 
I i i II 



rH O CN rH O rH O 
I ! I 



4J 



•H 0) 

-a > 

cd 

o 

U 
Ph 



o 



iH m 
o o 



m 
o 



m 
o 


m 
o 


m 
o 


m 
o 




n 


m 
o 


m 
o 


m 
o 


m 
o 


m 
o 


m 
o 


m 
o 


V 


m 
o 


m 
o 


V 
Ph 


V 
Ph 


m 
o 


V 
Ph 


m 
o 


m 
o 


V 


m 
o 


m 
o 


m 
o 


m 
o 


m 
o 


m 
o 


A 


A 


A 


A 


/» , ^ 


A 


A 


A 


A 


A 


A 


A 


A 


V 


A 


A 


V 


V 


A 


V 


A 


A 


V 


A 


A 


A 


A 


A 


A 


Ph 


Ph 


Ph 


Ph 


Ph Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


rH 


Ph 


Ph 


iH 


i— 1 


Ph 


rH 


Ph 


Ph 


rH 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 



o o 
o • 



o 
o 



3 


0) 




CO 


CO 


CO 


CN 






O 


rH 


rH 


00 


00 


CN 


00 


o 


CN 


o 


•<r 




m 


<r 


rH 


CN 




<r 


rH 


o 




co 




CO 


rH 




N 


CN 


rH 




in 




CN 


CN 


00 






rH 




CN 






00 




vo 


CN 


m 


O 




CN 




cn 


CO 


co 




m 


o 


o 


rH 




CN 


<r 


m 






CO 


vo 


m 


CN 


a> 


00 


00 


m 


8 


rH 


o 




o> 


o 


CO 


o 




m 




CO 


CO 


<r 


o 


CO 


rH 


w 


cd 


4H 




























































Ab 


> 


O 


rH 


rH 


rH 


rH 


rH 


o 


o 


o 


rH 


o 


rH 


rH 


o 


o 


CN 


rH 


o 


CN 


CN 


o 


CN 


rH 


o 


CN 


o 


o 


rH 


o 


rH 


rH 




0) 




o 


o 


m 


o 


o 


o 


m 


o 


o 


m 


m 


o 


o 


o 


m 


o 


m 


O 


o 


O" 


O 


o 


o 


m 


m 


m 


O 


o 


o 


o 




P* 

rH 


S3 


VD 




a> 


CO 


CO 


<j\ 


rH 


rH 


00 


CO 


m 


CN 


<r 


00 


a> 




rH 


CN 


o 


VD 


CN 


o> 


m 


co 


<r 


o> 


CO 


VD 


CO 


rH 




cd 


4H 


CO 


CN 


vD 




m 


a> 


VD 


o 


VD 


vD 


rH 




VD 


CO 


o 


m 


vo 


m 


rH 




00 




o 


o> 


CN 


o> 


rH 


CN 


m 


m 




> 


O 




m 


CO 


sr 


m 


VD 


VO 




<t 


m 


CO 




VD 




<r 


m 


VD 


CN 


CO 


VD 


<3- 


CO 


VD 


CN 


m 


m 


<r 




m 










CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 




H 
H 

3 



W 
Ph 

w 
p 



> 
W 
H 

a 



oooooooooooooooo 











CO 








CO 








CO 




CO 


CO 






M 




w 


CO 


rH 










w 


W 




i 






IS 


CO 










PJ 




rH 


O 


M 


Ph 


> 




> 


H 




PJ 


M 




rH 


CO 


H 


u 


CO 


a 


H 


CO 








rH 


U 


w 






o 




P 






o 


Ph 




P 






CO 


CO 


O 




§ 


W 


w 


w 


W 


§ 


Ph 


Pi 


Pi 


Pi 


H 


Ph 


M-) 


m 


m 


4H 


4H 


4H 


O 


o 


o 


O 


O 


O 


4H 


m 


m 


4H 


4H 


4H 



CN 




o o 



IH UH 



Hc^lcn<rlnvo^cocrlOHc^JfO<rlnvD^co^oHc^Jro<flnvD^co < ^o 

MrHMHrHr^rHrHHrHCNCNCNCNCNCNCNCNCNCNCO 



223 



w 
u 

52; 

w s 
o 

W CO 



Pm 
H 
P 



o 



> 



N CO 
> 



f*5 



U 
O 

CO W 

co pi 

<J H 

H O 
H 

CO 

CO W 



CO 

w 

H 



H 
H 



53 
o 



CO 

PM 

Bh 

pi! U 
CO o w 
H CO 

CO 

W O 
H H 

p$ 
4J W 

H 
P H 

<2 O 



o 

H 

H 

< 
P 

. > 
CO W 

PM ffl 
P H 



S5 
O 

W 



o 



2; 
o 

H 
H 
< 
P 
H 

> 

CO 
CO 

o 
u 



O 







O 
















O 


















O 












O 














it 
































































H 


H 


V 














m 


V 


m 


m 


m 


m 


m 


m 


m 


m 


V 


m 


m 


m 


m 


m 


V 


m 


in 


m 


m 


m 


m 


•H 


0) 




O 


O 


O 


O 


O 


O 


0 




0 


0 


0 


0 


0 


0 


0 


0 




0 


0 


0 


0 


0 




0 


0 


0 


0 


0 


0 


rO 


Rt 


PM 






























































CL) 






























































O 




V 


A 


A 


A 


A 


A 


A 


A 


V 


A 


A 


A 


A 


A 


A 


A 


A 


V 


A 


A 


A 


A 


A 


V 


A 


A 


A 


A 


A 


A 






H 


PM 




PM 


PM 


PM 


PM 


PM 


H 


PM 


PM 


PM 


PM 


PM 


PM 


PM 


PM 


H 


PM 


PM 


PM 


PM 


PM 


H 


PM 


PM 


PM 


PM 


PM 


PM 


PM 




O 
















O 


















O 












O 


















• 
















• 


















O 
• 












O 
• 














0) 




VD 


CO 




CO 


o> 


CO 




CO 






CO 






vO 


CM 


O 


00 




vO 


O 


H 


vO 


CM 






<r 


O 






CO 




-U 


CM 


O 




CM 


00 


CM 


<f 


in 


CO 


O 


O 


00 


CM 


00 


CO 


m 


vO 


00 


CM 


O 


<r 


h- 


00 


O 




00 


o> 


vO 


o> 


o> 


H 


































































*H 


CM 


O 


0 


O 


H 


O 


O 


0 


CM 


O 


O 


0 


O 


O 


O 


0 


O 


CM 


O 


O 


1— l 


0 


1— 1 


CO 


1— 1 


O 


O 


1— 1 


0 


0 


> 


O 


1 


1 


l 








I 




I 


1 


I 




! 


1 




1 


1 


1 


1 


1 






1 


1 


I 


1 




I 




1 






LP) 


































H 






m 
























O 


































O 






0 




















it 


















































H 














H 


H 


V 












m 


m 


m 


m 


m 


m 


m 


m 


m 


m 


in 


V 


m 


m 


V 


m 


m 


O 


m 


m 


m 


m 


m 


in 


•H 


CU 




O 


O 


O 


0 


O 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 




0 


0 




0 


0 


O 


0 


0 


0 


0 


0 


0 






PM 




























































































































rO 
O 




V 


A 


A 


A 


A 


A 


A 


A 


A 


A 


A 


A 


A 


A 


A 


A 


A 


V 


A 


A 


V 


A 


A 


V 


A 


A 


' A 


A 


A 


A 


v< 




H 


PM 


PM 


PM 


PM 


PM 




PM 


PM 


PM 


PM 


PM 


Pm 


PM 


PM 


PM 


PM 


H 


PM 




H 


PM 


Pm 


PM 


Pm 


Pm 


Pm 


PM 


PM 


PM 


PM 




O 
































O 




O 
























































O 



























•H 
> 

O 
0) 

0 

53 



OiHOOiHOOCMiHOiHOiHcOfHOOOtHiH 



0) 


























4-> 




























CU 




rH 






<r 


CM 


<r 




tH 


tH 


CM 


H 




N 


CM 


CO 






tH 


0 


00 


00 


<r 


00 


O 


H 




H 


<r 


CM 


CM 


m 


CO 


m 


m 


0 


<t 


CO 




<H 






















& 


> 


O 


CM 


0 


0 


O 


1— 1 


0 


O 


0 


rH 


O 






























0) 




O 


0 


0 


m 


m 


O 


m 


0 


O 


m 






P 
























tH 




vO 


CO 


H 




<r 


CM 




H 


H 


CO 




cd 




ON 




00 


a\ 


<r 


O 


\0 


0 


CM 


m 




> 


O 


CM 


vo 


VO 


VO 


m 




\0 




CO 


vO 








CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 



vo 00 00 o\ m 00 

OvOCnHOOOOO 

mvo^rr^^rvo^roN 



CM in tH cm vo o> vo 
fN Q> 0> 00 CM 



cm 00 cm o 



O(N0>cr\<r<r<rcr\ 



CM CO CM VO 

00 co vo 



ooomomoommoooominominm 

Hc^oo<ro<-HoooococoincocovoiniHOr^in 
cm vo^vo vococovo^ooooOr^coa^oo^romo 
vo<rvovoinvovocMinr^<TincocMcoinvovoirico 




M_l M_l M_l M_l M_l M_l M_l 
OOOOOOO 

M_| M_| M_| M_| q_| q_| LH 



•3 

C 
•H 
4J 

c 
o 
u 



0) 

O rO 

O M 
53 cd 
> 



Hf^ln<^lnvo^ooc^OHNfn^lno^^JC^OH^lcostlOvo^ooc^O 
cococococococococo<r<r<r<rvd-vd-vd-^^v^Lnini^ 



224 



•H 

rH rH 
•H 0) 
> 

o 

PM 



t— I 
O 



H 

o m 
o o 



V A 



PM PM rH 
O 

o 



rH 

o in m m 
o o o o 



V A A A 





S3 






0) 


4J 




H 


H 




o 




O 










r-. 


CO 


CN 


o> 


o 




CO 






t— 1 


m 














H 






cd 


o 


<r 


H 


CO 


CO 


H 










> 




l 










H 




<~\ 


















EAN D 


COMP 


(19R 




4J 








H 
O 






S 


CO 






*H 




rH 






H 






PM 


o 




H 


!— 1 


o 


m 


V 


O 


m 






H 




•H 


0) 


o 


o 




o 


o 


o 


RO 


CT 




cd 


ev 






PM 


» 




CO 


o 


w 








V 


A 


V 


V 


A 


H 




CO 




o 














CO 












PM 


PM 


H 


PM 


PM 


w 


o 






PM 








O 




H 


RI 


O 
H 












O 
• 






•U 


w 


H 








































H 


























CD 


















O 




4J 


















Pi 


> 




CD 




00 


<r 


CO 


vO 


O 


CO 


W 


iH 




CO 


ON 


CO 


CN 


<r 


O 


W 


W 




O 


H 




m 


CN 


CN 


o 


O 




PM 








m 














PM 






> 


o 


<r 


H 


CO 


m 


H 


3 




H 



















A A 



CN 



0) 

cd m 
> o 



m m o o o o o 

o oo cr> <r i— ) co cn 

N rl O M O N CM 

in <r o> o> n <r <r 

rH CN rH rH CN CM CN 



H O 



0) 
rH 

cd 

*H 
U 

cd 
> 

m 
o 

0) 

3 



0) 
CO 









CO 


CO 


H 




CO 


CO 


CO 


4J 






B 


4J 


4J 








d 


0) 




jg 




to 


00 tH 


H 


W) 




00 *H 


•H 






iH 


•H 


iH 


0) 


CD 


CO 


X 


0) 


0) 


0) 


& 


& 




0) 


& 


JS 


E5 






M 










rH 


CN 


O 


d 


CO 


CN 


rH 


1 


J 


ES H 


m 


m 


m 


m 


M-l 


m 




o 


o 


o 


o 


O 


o 


O 












u 






0) 






0) 


CD 


CD 
















■§ 




■§ 


0 


■a 


■§ 




y 


I 
















1 






1 


rH 


rH 


rH 


rH 


rH 


rH 


rH 


cd 


cd 


cd 


td 


cd 


cd 


cd 


4J 


4J 


4J 


4J 


4J 


4J 


4J 


O 


O 


O 


o 


O 


O 


o 


H 


H 


H 


H 


H 


H 


H 



0) 

M-l rH 
o rO 

cd 

• 'rl 
O M 

& cd 
> 



h cm co m vD N 
\D vO vO vO vO vO vD 



0 

ERIC 



225 



tH tH 



cd <u 

O 

u 

Ph 



m 
o 



LT) 

o 



inmiriiriiriioiniriioiriiriHioioiriiriin v m in v in r i m m m m m m m 
ooooooooooooooooo oo ooooooooo 

AAAAAAAAAAAllAAAAAVAAVAllAAAAAAA 

PmPmPmPhPmPmPmPmPmPmPhPhPhPhPhPhPhiHPh 

O O 



CD 

•H 

4J 

■ C3 
O 

o 



w- 
a 

Ph 

M 
P 

o 

CO 
H 
CO 

w 

H 



CO 
W 

> 



P 
W 
H 



ON 






0) 










in 


o 




m 


m 


m 


CO 


CM 


tH 






00 


00 














r— j 






o 


CM 


in 


CO 




CO 












co 




, — I 


o 












co 


vD 


CM 


00 


co 


r— j 


00 




r— ) 


00 


tH'CM 


<o 


00 








00 


CM 










H 






































































Va 


o 


O 
I 


o 


o 
j 


o 
i 


o. 


© 


o 


o 
t 


o 


o 

j 


o 
I 


CM 
| 


o 


o 


o 


o 


r— | 

( 


CM 
J 


r— | 


r— j 


CM 


rH 


CM 


o 


o 


r— j 
J 


o 


o 


o 


o 






































I 










| 




















1 


s 


P^ 




































































o 


o> 




































































u 


H 




>> 


























m 
o 












m 
o 




m 
o 


m 
o 




m 
o 






m 
o 










w 






4J 
































































Pi 


2! 




mH 


































































O 




tH 


rH 


m 


in 


m 


in 


m 


in 


m 


m 


in 


m 


m 


V 


in 


in 


in 


m 


in 


V 


m 


V 


V 


m 


V 


in 


m 


V 


in 


in 


m 


in 


O 


H 




*H 


0) 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 




o 


o 


o 


o 


o 




O 






o 




o 


o 




o 


o 


o 


o 




H 






5 




































Ph 




Ph 


Ph 




Ph 






Ph 










o 


a 




cd 


































































w 








A 


A 


A 


A 


A 


A 


A 


A 


A 


A 


A 


V 


A 


A 


A 


A 


A 


V 


A 


V 


V 


A 


V 


A 


A 


V 


A 


A 


A 


A 




CO 




o 
































































o 










Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


tH 


Ph 


Ph 


Ph 


Ph 


Ph 


tH 


Ph 


tH 


tH 


Ph 


tH 


Ph 


Ph 


tH 


Ph 


Ph 


Ph 


Ph 


H 






Ph 


























o 












O 




O 


O 




O 






O 






Pi 


o 




































































w 


H 




































































H 


H 




































































f— 1 


< 




































































pi 










































































a) 

4J 


































































W 


> 




0) 




00 


CO 


CO 


<f 


O 


CT\ 




CO 






m 


tH 


tH 


O 




CO 


tH 


tH 


vD 




00 


CO 


00 




CO 


rH 


CO 


CO 


m 


O 




PJ 


H 




N 


o 


<r 


o> 


v£> 


o 


<t 


CM 


in 


00 


m 


m 


CM 






m 


m 


tH 


tH 


CO 






CO 


rH 




CM 


00 


ON 


CM 


co 


<r 


2 




O 


T— 1 






CM 


m 


O 


o 


O 






CM 


.vO 


CM 


CM 


CM 


vD 


m 


rH 


C^ 


<J- 


CM 


O 


o 


CM 


<i- 


CM 


O 


CM 


00 


m 


CM 


a\ 


Q 


W 


CO 


cd 
































































H 






> 


O 


o 


o 


o 


O 


o 


o 


o 


O 


O 


O 


o 


CM 


o 


O 


o 


o 


tH 


CM 


tH 


CM 


CM 


O 


CM 


rH 


rH 


CM 


O 


o 


O 


o 




H 


< 




































































S3 




































































rr 


o 




































































> 


w 




ue 


S3 


o 


o 


m 


O 


o 


o 


m 


O 


in 


O 


o 


O 


o 


O 


m 


o 


O 


m 


m 


O 


O 


o 


m 


m 


m 


O 


o 


o 


o 


m 








tH 








<r 




CO 


CM 


00 


tH 




CM 






tH 


CM 


m 


ON 


CO 


m 




o 




CM 


m 


CO 


00 


CM 






tH 


CO 








cd 




m 


C7\ 




CM 


CO 


CO 


CM 


o 




tH 




00 


O 




CO 


o 


O 


v£> 


m 


tH 




CO 


CO 


C^ 


CO 


rH 


CO 




O 










> 


O 




vO 


m 






r^> 






vO 


vD 


vD 


CM 






vD 




<f 


CO 




<f 




vD 


tH 


m 


<r 


CO 


m 




r** 




o 










CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 






O 


>-* 


O 


Ph 


of 


of 


of 


of 


of 













14-j 14-J 14-J 14-j 14-j 14-j 

o o o o o o 



oooooooooooo 



M-l M-l M-l 

o o o 



o o o 



0) 

M-l tH 
O 

cd 

o u 

25 cd 

> 



H^cn^invDNCO^OHNrO^inONoO^OrlCNrO^inONCO^O 

iHt-ItHrHtHiHiHiHiHiHCMCMCMCMCMCMcMCMCMCMcO 



0 

ERIC 



226 



w 
u 

w 



o 

CO 
H 
CO 

w 

H 



CO 

w 
> 



p 
w 

H 



55 
o 



CO 



o 
W co 



p 
o 

o 
o 

H 

W 
H 
H 



cn 



53 
O 
H 
H 
O 
W 
CO 

25 

o 

H 
H 

<! 
P 



CO 

> 



CO 
CO W 

CO pQ 
W < 
H H 



H 



W > 
P 

as 



O 



O 

CO W 

HI ptf 

H O _ 

H ft| H 



CO 

2S 
O 
H 

p 



3 

> 

00 
CO 

o 
o 



O 



•H 0) 
4=1 > 

cd a) 
4=- >J 
o 
M 

PM 



LO 

o 



LO 

o 



m lo 
o o 



0) 
4J 



4=1 

cd 

cd 
> 

O 



oo oooooooo ooooo o oooooooo 

• • pH P-l pL(»pL|pL| pL( 

AAVAAAAAAAAVAAAAAV AV VAAAAAAAAV 
PhPhHPhPhPhPhPhPhPhPhHPhPhPhPhPhHPhHH 

o o o o o o 

• o • • • o 



0) 


4J 


CM 


tH 


sr 




o 


CM 


CO 






LO 


H 


LO 


H 








CM 




CO 


CO 


H 


CO 




LO 


CO 


iH 


LO 




LO 


cn 






tH 


cn 


cn 


CO 


o 


LO 


tH 






CO 






vO 


CM 


CM 


LO 


CO 






H 


vO 


H 




CO 


CM 


LO 


CO 


<• 


CO 




iH 


m-i 






























































Va 


o 


1 


o 
1 


CM 
1 


H 
1 


o 


O 


O 
1 






— I 

1 


o 
l 


CM 
1 


O 
1 


tH 
1 


tH 
1 


o 
I 


tH 
1 


CM 

! 


o 


CM 
1 


CM 


H 


O 


H 
1 


H 
1 


H 
1 


o 
1 


H 
1 


o 
1 


CM 
1 




























LO 












LO 




LO 


LO 










LO 








H 


>> 


























o 












o 




o 


O 










O 








O 


it 


































































tH 


in 


lo 


LO 


lo 


lo 


lo 


lo 


LO 


LO 


LO 


LO 


V 


LO 


LO 


LO 


LO 


LO 


V 


LO 


V 


V 


LO 


LO 


LO 


LO 


V 


lo 


LO 


LO 


V 


•H 


0) 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 




O 


O 


o 


O 


O 




o 






o 


o 


o 


O 




o 


o 


O 






5 




































PM 




PM 


PM 










PM 








PM 


cd 




































































A 


A 


A 


A 


A 


A 


A 


A 


A 


A 


A 


V 


A 


A 


A 


A 


A 


V 


A 


V 


V 


A 


A 


A 


A 


V 


A 


A 


A 


V 


o 




p-l 


Ph 


Pn 


Pn 


Pn 


Ph 


Pn 


Ph 




PM 


PL. 


tH 


PM 


PM 


PL. 


PL. 


PL. 


H 


PM 


H 


H 


PM 


PM 


PM 


PM 


H 


PM 


PM 


PM 


H 


Pn 


























O 












O 




O 


O 










O 








O 




























• 




































o 



3 


0) 




iH 


CO 


o 


o 




LO 


CO 


CO 


<r 




O 


<r 


VO 


CM 


CM 


00 


cn 


o 


O 






cn 






CO 


\D 


CO 


CO 


CO 


LO 


H 




N 


iH 


LO 




co 


o 




CO 




cn 




<r 


m 


CM 


cn 


CM 


CO 


CM 


LO 


CO 




H 






cn 


CM 


O 


o 


CM 


cn 


H 


O 


H 




iH 


O 




CM 


o 


o 


CM 


<r 




O 


CO 


CM 


CO 




CM 


<r 


cn 


<r 


O 


LO 


H 


o 




CO 




CO 


CM 


CO 


O 




CO 


cd 


«H 






























































Ab 


> 


C 


; 


H 


H 


H 


o 


o 


O 


o 


O 


tH 


o 


CM 


o 


O 


tH 


fH 


H 


CM 


H 


CM 


CM 


o 


o 


H 


H 


CM 


O 


H 


O 


CM 




0) 




o 


O 


lo 


in 


o 


o 


LO 


o 


O 


O 


o 


LO 




LO 


LO 


LO 


LO 


LO 


LO 


LO 


O 


o 


o 


O 


LO 


LO 


LO 


LO 


LO 


LO 




H 


P 


vO 


cn 


cn 


LO 




o 


<r 


cn 




LO 




LO 


CO 


cn 


•vf 


CM 


cn 


CO 


CM 


CM 




H 




CO 


\D 


LO 


o 


CO 


CM 


© 




cd 


<H 


iH 


in 


o 


CM 


CO 


CO 


o 


cn 




CO 


LO 




cn 


LO 


tH 


vO 


cn 


LO 


H 


CO 




CM 




CM 


H 


O 


cn 


o 




CO 




> 


O 


LO 


LO 


CO 


LO 












LO 


\D 


CM 




vO 


LO 


<r 


CO 


co 


LO 


CO 


<r 


r>. 


LO 


LO 


CO 


CO 


vO 


LO 




o 








CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


J 


:n 




q_| q_| q_| q_| 
O O O O 



O O 



H 




CO 


21 




O 


»J 








►J 




SI 


W 


H 




H 


CO 


O 


CO 


P 


CO 


o 


M-l 


M-f 


M-f 


M-l 


O 


O 


O 


O 




n 
g 

•H 
•U 
CJ 
O 

o 



M-l iH 
O 43 

cd 

•H 
M 
cd 
> 



o 



hcmco<j-lovo r^cocnOHcMco<-LOvor^cocnOHcM co<j-LOvor^cocnO 

COCOCOCOCOC^COCOCO<"<-<"^<-<"<-<-<f<-LOLOLOLOLT^,L^ 



227 



9 

ERIC 



4J 













H 








>> 










O 








4J 










• 








•H 




H 














i— 1 


H 


O 


m 


m 


V 


m 


m 


m 


•H 


0) 


O 


o 


o 




o 


o 


o 




H 








PM 








■§ 






















II 


A 


A 


V 


A 


A 


A 


O 






















PM 


PM 


PM 


H 


PM 


PM 


PM 


PM 










O 
















O 









0) 


+J 




00 


-3- 




o 


00 


00 






CO 


CM 


00 


00 


o 


o 


CM 


H 


M-l 
















td 


O 


CO 


d 


d 


Cn] 


H 


d 


H 


> 




1 


1 


l 








I 



4J 
•H 

i— I i— I 

•H CU 

id dJ 
o 

U 
PM 



o m m o m m m 
o o o O o o o 



V A A V A A A 

f^l 



0) 
4J 

3 a) 

O H 

cn cd M-i 

*3 > o 



vO M fO vO O 00 vO 

o> cm o oo o r> in 

CO H 0> vO O M N 

CO O O CO H O H 



Pi! 



0) 
H 

td M-t 
> o 



o m o o o m m 

cm vo lo h H m CM 

00 o O ON O vO 00 

CO N ^ O fN O CM 

iH CM CM CM CM CM CM 

































0) 
















cn 














4J 
































0) 


CO 










CO 


CO 


H 






cn 


CO 


cn 


4J 


4J 




g 




4J 


4J 


4J 






P3 


0) 


H 






& 


on 




•H 


H 




W) 




W> 


•H 


•n 






td 


•H 


•H 


•rl 


0) 


CD 


cn 




•H 


0) 


0) 


0) 


& 


& 










& 


& 










td 








H 


CM 






> 


CO 


CM 


H 


1 


1 


1 




4-1 


M-l 


M-l 


M-l 


M-l 


4-1 


M-l 


M-l 


o 


O 


O 


O 


O 


O 


O 


O 








U 












0) 


0) 


0) 


0) 


0) 


0) 


0) 


1 


















■§ 




! 


1 




•§ 


■a 


























S3 






i 




al 


al 


al 


al 


al 


al 


al 




4J 


4J 


4J 


4J 


4J 


4J 


•u 




O 


O 


O 


O 


O 


O 


o 




H 


H 


H 


H 


H 


H 


H 



0) 

M-l H 
O ,£1 

..3 

O M 

Sz; td 
> 



H cm co <t m vo 

vO vO vO vO vO vO vO 



228 



•rl 

i— I H 
•H 0) 

a) 

o 
u 

PM 



m in 
o o 



O 



O 



LO 

o 



LO LT) 

O O 



LO 

o 



LOLTlV V LT) LT) LT) LT) LT) LT) V O LT) LT> V LT) LT) LT) LT) V LT) LT) V V LO LO LO LO LO V 

oo oooooo ooo oooo oo ooooo 

• • PM pM pLi • • • pL< • •. CL< ...pupD pL| 

AAVVAAA.AAAV vAAVAAAAVAAVVAAAAAV 

oo oo ooo o 



CD 
J 
C 
■H 

o 
o 



w 








u 








S3 


5 






w 


o 






Pi 


CO 






c=a 


H 




















> 


H 




PS 




P 











o 


H 




U 


s—' 


i 


00 


S3 




PM 


O 






H 




o 


H 


o 


Pi! 


a 




O 


[=3 


CO 




00 


H 


S3 




CO 


O 




W 


H 


o 


H 


^ 


M 




W 




4J 


H 


S3 




H 






Pi 






O 












Pi 




00 


W 






P-i 






P-i 
















> 








00 


o 


N 


> 








W 



as 

o 

00 W 

00 3 

<J H 

PC pi 

H O 

00 
00 W 
H >J 

oo pq 
w < 

E-f H 



H 
H 



vo 
O 



o 

M 

H 
<J 
P 
H 

> 

00 
00 
O 

Pi 
o 

w 
w 

H 
O 



•H Q) 

rO J> 

o 

PM 



^^HrH^OOO^inO^CNCO^OCMCOmmvDvDHO^CNNHinO 

OONNrlOOHOrlMmOOMHHOOMOOMMOHOOrlCM 
I I I I i i III 



lo 
o 



ininmiriuiiriiri^^uiiriOiriiri v 

oooooooooooooo 



LT) LT) LT) LT) V LOLOLOLOLOLOLOLOLOLO 

oooo oooooooooo 

PM • • ■• * PM 

AAAAAAAAAAAvAAVAAAAVAAAAAAAAAA 

pMpMpMpMpMpMpMpMpMpMpMpMpMpMHpMP^ 

O O 



3 


0) 




H 


H 


I— I 








o 


o 


CO 






CO 




O 


o\ 




CM 


CO 




CM 




CO 


m 


<r 




co 


CM 


LO 


CO 




1— 1 




N 


O 


CO 


LO 




<- 


00 


CM 


1— I 


LO 


H 


00 


CO 


LO 




o 


•<r 


CM 


CM 


co 




H 


CO 


CO 


00 <" 


o 


O 


vO 


00 


LO 


o 


i— i 




00 


vO 


a> 




<J- 


CM 


CO 


o 


o\ 


vO 




CO 


CM 


vO 




CM 


<r 


vO 




<r 


O 


<r 


vO 


CO 


O 


LO 


LO 


LO 


LO 


CO 


CO 




m 






























































Ab 


> 


o 


o 


H 


H 


i— I 


i— ! 


o 


o 


H 


o 


H 


H 


CO 


O 


O 


CM 


i— 1 


i— I 


o 


O 


CM 


O 


i— 1 


H 


H 


O 


H 


O 


o 


H 


H 




0) 




LO 


o 


o 


o 


o 


o 


LO 


o 


o 


LO 


LO 


o 


o 


o 


LO 


O 


LO 


LO 


o 


LO 


LO 


LO 


LO 


o 


u~\ 


m 


LO 


O 


O 


LO 




i— i 


P 


o 


i— 1 


00 




CO 


<r 


CO 


<- 


<r 


CO 


vO 




iH 


CM 


CM 






o 


CM 




vO 


ON 


CO 


LO 


vO 


<r 


ON 


CM 


vO 


LO 






m 




a> 


00 


VD 


LO 


o 


o 


vO 


CO 




a\ 


H 


O 


vO 


o 


CM 


CO 


CM 


<J\ 


vO 


CO 


ON 


H 


CO 


CM 




CM 




i— I 


LO 




> 


o 


LO 


<■ 


CM 


<T 


LO 


rs 




vO 


LO 




co 


o 


r*. 


vO 


CO 


LO 


<■ 


vO 


LO 


CO 


r^* 


CO 


CO 


<r 


r-. 


<r 


vO 


vo 


LO 


CM 








CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 




oocooocooooooooooooooooooooo 



LM 

o n 

LM 00 



0) 

LH i— I 

°-8] 

. -HI 
C U 
V5 as 



rlrlrlrlrlrlrlrlrlrlCMMMMMMMMNNn 



229 



4J 
•H 

rH rH 
•H 0) 
.G > 

a) 

O 

u 

Pn 



o 



o 



m m 
o o 



m m io lo v 
o o o o 



m m m m m m 
o o o o o o 



v m m in m m m m v mmv v n n m n n n 
ooooooo oo oooooo 

Ph 



AAAAVAAAAAAVAA 



AAAAVAAVVAAAAAA 



O O O O O 



-a 

o 
c 

•H 

•U 

C 

■ O 
O 



53 



O 

o 

CO W 

co X 
< H 



53 
o 

H 
H 

« & a 

H O 

H p4 



CO 
CO 

o 
o 



> w 

»J O 



CO 
CO W 
H >J 
CO PQ 
W < 
H H 



23 23 
S O 



w 


O 






0) 


4J 


m 


on 


00 










O 


iH 






00 




00 


00 


o 


00 




00 




00 




CO 


m 


00 


CO 


CO 




CO 






CO 










tH 




m 






CM 


CO 


CO 




iH 




iH 


CO 




CO 


o 


o 


CO 


00 


CO 




CM 


m 


o 




n 


CM 




CM 




w 


H 






H 


IW 








































































O 


iH 


o 


H 


H 


H 


o 


o 


rH 


o 


iH 


O 


CO 


o 


o 


iH 


o 


tH 


o 


o 


CM 


o 


O 


CM 


CM 


o 


o 


rH 


o 


rH 


o 










> 




1 










1 


I 




I 


















1 


1 




1 


1 


I 


[ 


1 






I 






M 


Ph 






































































Q 








































































o 


rH 




































































13 


u 






































iH 










m 






m 
















ES 








































o 










o 






o 


















CO 






4J 
































































a 


PH 


o 




•H 


























iH 








































p 


M 




tH 


tH 


m 


m 


m 


m 


m 


m 


m 


m 


m 


m 


m 


O 


m 


m 


V 


m 


m 


m 


m 


V 


m 


m 


V 


m 


m 


m 


m 


n 


m 


n 




o 


H 




•H 


0) 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


O 


o 


o 




o 


o 


o 


o 




o 


o 




o 


o 


o 


o 


o 


o 


o 


o 




O 






> 
































































o 


w 




•§ 


0) 






























































CO 




CO 








A 


A 


A 


A 


A 


A 


A 


A 


A 


A 


A 


V 


A 


A 


V 


A 


A 


A 


A 


V 


A 


A 


V 


A 


A 


A 


A 


A 


A 


A 


H 


25 






O 
































































CO 


o 


23 




M 




PM 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Pm 


Pm 


Ph 


Ph 


Ph 


tH 


Ph 


Ph 


Ph 


Ph 


rH 


Ph 


Ph 


rH 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


w 


H 


o 




Ph 
































o 










O 






O 
















H 




H 




































o 


































w 


H 




































• 
































4J 


H 


<J 






































































H 


P 






































































Pi 


>J 






































































O 




CL) 








































































■U 




































































W 




0) 






iH 


o\ 


o 






H 


*<r 












00 


CO 


CO 


CO 


CO 


CO 


00 




o 




<r 


rH 




CO 


rH 


CM 


CM 


CO 


w 




iH 




N 




o> 


CM 


CM 


CO 


<r 


KD 


CM 


o 


CO 


CO 




m 




CM 


00 


o 


<r 


CM 


tH 


CO 


<r 


o 


CO 


o> 




m 




00 


CM 


w 


Ph 


w 


O 


i— i 




o 


n 


\D 




m 


CM 


CO 


O 


o 


<r 




vO 


iH 


o 


VJD 


o 




in 


tH 


<r 


o 


o 


o 


CO 


CM 


CO 






o 




p 


Ph 


w 


W 


td 


m 






























































P 


H 




o 


iH 


iH 


iH 


iH 


iH 


O 


O 


tH 


tH 


H 


o 


CO 


O 


o 


CM 


tH 


tH 


o 


O 


CM 


o 


rH 


CM 


rH 


•o 


rH 


o 


o 


tH 


o 


> 




22 






































































CO 


O 




































































N 


> 










































































w 




0) 




m 


O 


O 


m 


m 


in 


O 


O 


O 


o 


o 


o 


m 


m 


O 


O 


m 


o 


m 


m 


o 


m 


O 


O 


O 


O 


o 


o 


O 


o 


@ 


w 








p 
































































>J 






i— i 




o 




iH 




tH 


CO 


o> 


CO 


O 






o 


m 


CM 


m 


H 




u » 


CO 


CO 


in 


00 




m 




o 


«<r 


CO 




o 


H 


Q 






cd 


m 


iH 










o 




v£3 


CM 




m 


<r 


rH 


CO 


tH 


m 


00 


CO 


rH 




CO 


CO 


CM 


CO 






o> 




00 


in 


<I 


Q 






> 


o 


m 


<r 


co 




in 






v£> 


W) 


<T 


m 


o 






CO 


m 


<r 


v£3 




CO 




<r 


CM 


<r 






m 


KD 


in 


in 


M 


H 


CO 








CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 



•H 
U 

> 
o 

0) 
25 




H_l g_| tH m U-| iw 

^ ^ ^ ^ ■> s 



0) 

HH rH 
O ^5 

. H 

O H 

> 



HCMCO^tlT vjd M»(^OHCMf0<rm^N00CnOHCMn<r^^N00^O 

cococococococococo<r<r^^^s^^^^N^ininu^ 



230 



•H 0) 

O 

u 

PM 



m m o m m m v 
o o o o o o 

PM 

A A V A A A V 

P-i P-i P-i P-i P-i P-i H 
O 
O 



0) 




m 


C^ 


H 


H 


H 


On 


00 






vO 


<f 


O 


m 


m 


O 


00 


H 




















O 


H 


H 


<f 


H 


H 


H 


CM 


> 




.1 
























m 






H 












o 






O 


4J 


















•H 








H 










H 


H 


m 


m 


O 


V 


m 


m 


V 


•H 


0) 


o 


o 


O 




o 


o 




rO 










PM 






PM 


nj 


















rO 




A 


A 


V 


V 


A 


A 


V 


O 


















u 




PM 


PM 


PM 


H 


PM 


PM 


H 


PM 










O 






O 


















O 



•u 



CO 



0) 


















4J 




















0) 




o 


h <r 




o 




H 


H 




N 


H 


00 


vO 


o 


CM 




O 


H 




CO 




CM 


o 


H 




CO 


cd 
















3 


> 


O 


H 


H CO 


CM 


o 


H 


CM 








m 


o o 


o 


o 


O 


m 


























H 


CM 1^ 


cn 


00 


m 






nj 


m 


O 


CO vO 


m 


CO 




CM 




> 


o 


<r 




m 






o 








CM 


CM H 


CM 


CM 


CM 


CM 

































(D 
















CO 














4J 
















8 


CO 










CO 


CO 


H 






CO 


CO 


CO 


4J 


4J 




B 


CD 


4J 


4J 


4J 




xi 


ti 


CD 


H 








00 


00 «H 


H 




b0 


00 


b0»H 


•H 






cd 


•H 


•H 


•H 


CD 


CD 


CO 


X 




CD 


CD 


<D 


& 


J5 




(D 


H 




& 








U 












H 


CM 




a 


> 


CO 


CM 


H 


1 


1 




H 




iw 


IW 




IW 


IW 






o 


o 


o 


o 


o 


o 


O 


O 


0) 






u 
















CD 


CD 


CD 


CD 


CD 




















■s 


■§ 


"§ 




"§ 


■i 






















1 


1 








1 


1 




H 


H 


H 


H 


H 


H 


H 




nj 


nj 


nj 


nj 


nj 


nj 


nj 




4J 


4J 


4J 


4J 


4J 


4J 


4J 




O 


O 


O 


O 


O 


O 


O 




H 


H 


H 


H 


H 


H 


H 



CD 

m H 

O rO 

nj 
. *H 
O H 

JZJ nj 
> 



h cm ro <r m vo 

vO vO vO vO vO vO vO 



0 

ERIC 



/ 



231 





















W 












W 


CO 






f— i 




Tr. 
MH 




CO 


h- 1 


< 


o> 


1— 1 




, 1 






•> — > 


25 


/ — \ 






u 




w 




^ — > 




CO 






p_, 


H 


C=-i 


P 


u 


o 




w 




§ 


CO 


CO 






H 






CO 




o 


w 




f— 1 


H 


M 






Ph* 




4J 


W 






H 


f— | 


P 


I— 1 


[>( 






f— | 


<! 


o 


£_| 






CO 


CO 


PA* 




W 


w 




P 


pL, 






Ph 


w 


a 












N 


CO 






> 


o 


P 






w 


PS 1 


w 


H 


w 




3 


£s 






o 




U 






o 




CO 


CO 


w 




CO 




25 


< 


H 


O 






H 


p 






H 


o 


S3 


H 




P 


!5 




H 




CO 




CO 


w 




H 


hh* 


> 


CO 






w 




CO 


H 




CO 








P 


1 


§ 






u 








W 




w 


2S 






H 






H 


hh* 






53 








o 



25 
O 













r r\ 

u ) 






, | 


































in 














































































4-J 

•<—i 
pi 




rH 


rH 


rH 




, | 








, | 


, j 


, | 


, | 


, | 










j j 


i 1 




,_j 




r ■} 
















rH 


rH 


O 


O 


o 


V 


O 




V 


m 


O 


O 


O 


O 


O 


LO 


O 


O 


V 


O 


O 


O 


O 


O 


o 


V 


O 


O 


O 


O 


O 


O 


•H 


0) 


o 


O 


o 




o 


O 




o 


O 


O 


O 


O 


O 


O 


o 


o 




O 


O 


O 


O 


O 


o 




o 


O 


O 


O 


O 


O 


rQ 
































































cfl 
































































.-Q 




v 


V 


v 


v 






V 


v 


V 


V 


V 


V 




v 


v 


v 


V 


V 


V 


V 


V 


v 




v 


V 


V 


V 


V 


V 


O 
r-i 




Ph 


Ph 


Ph 


rH 


_ 

Ph 


Ph 


i — ! 


Ph 


f-H 


_ 

Ph 


Ph 


Ph 


_ 

Ph 


_ 

Ph 


Ph 


p 


i — ! 


Ph 


Ph 


Ph 


Ph 


Ph 


_ 

Ph 


rH 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 






/ — i 


























































































* 




























0) 


4J 






00 


<T 


rH 




CO 


00 


m 




rH 


CM 










CM 


rH 


CM 


rH 


<D 


CO 


On 




O 




<r 


O 


00 


00 






CO 




CO 


CM 


o 


00 










CO 


, | 






<D 


CO 


00 








CO 




CM 


, — 1 




CM 




CO 


in 




rH 


1 1 j 






























































Cfl 


O 




<r 




CM 




rH 


CM 


rH 




in 






CO 


rH 


CO 


in 


CM 




VO 


<r 


in 


<D 




CM 






00 


<r 


<r 


rH 


t> 




1 


i 


i 
i 


i 
1 


i 


| 




| 


i 
1 




1 






| 


1 




i 
i 


l 




i 






l 


1 


i 




i 


i 


i 


rH 
1 
















m 


rH 


































m 
























o 




o 


O 


O 
































o 














4J 
































































tH 




rH 


rH 


rH 




rH 








rH 


rH 


rH 


rH 


rH 




rH 


rH 


rH 


rH 


rH 


rH 


rH 


rH 


rH 




rH 


rH 


rH 


rH 


rH 


rH 


rH 


rH 


O 


O 


O 


V 


O 


V 


V 


V 


O 


O 


O 


O 


O 




O 


O 


O 


O 


O 


O 


O 


O 


O 


V 


O 


O 


O 


O 


O 


O 


•H 


0) 


O 


O 


O 




O 








O 


O 


O 


O 


O 


o 


O 


O 


O 


O 


O 


O 


O 


O 


O 




O 


O 


O 


O 


O 


O 




> 








Ph 




Ph 


Ph 


Ph 














































Cfl 


0) 
































































rH* 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


A 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


o 
n 




Ph 


Ph 


Ph 


rH 


Ph 


rH 


rH 


rH 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


rH 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 










O 




O 


O 


O 
































O 




























O 

















































0) 

u 

rH 2 N 

O rH 

W CU 14H 

43 > O 
< 



0) 
rH 

> O 



M-l rH 



O 
21 



00 


<r 


m 


O 










00 


ON 


CO 


CO 


CO 


CM 


co 




CO 




rH 


<D 






00 


00 


m 








o 


CM 




O 




m 


o 


<D 


ON 




m 


ON 


CO 


o> 






o 


rH 


o> 


CM 




00 


<- 


CM 


CO 




CO 


00 


co 






00 


m 




<d- 


CM 


o> 


CO 


rH 


CM 








rH 


00 




cr> 




CO 


00 






<r 


CM 


r^> 


rH 


o 


rH 




<t 


<r 


o 


<r 


m 




CM 


CO 


CM 


CO 


CM 




m 


<D 




CO 


rH 


CO 


m 


CO 


m 




<r 


LO 




o> 


CM 


00 


<r 






<r 


o 

rH 


m 


o 


o 


m 


o 


m 


O 


O 


o 


o 


O 


m 


O 


O 


o 


m 


m 


o 


O 


o 


o 


o 


o 


m 


o 


m 


m 


m 


o 


m 


\D 


m 




o 




CM 


o> 


CO 


m 






CO 


CO 




IT) 


rH 


rH 


rH 


m 


CM 


CM 


00 


vO 


m 




r^> 


CO 


CM 


CM 




CM 




00 


<r 


o> 


00 


CM 


m 


m 




m 


<r 


<r 


CO 




CPs 


m 




vD 


<r 


CO 


00 


CO 


rH 




CM 


rH 


CO 


in 




<T\ 






<r 


o 


<f 


CO 


m 


CO 




rH 




rH 


m 




m 


rH 




CM 


00 




o 


CO 


m 


CO 


O 


O 


o 




rH 


rH 


rH 




CM 


CM 


CM 


CM 


CM 


rH 


rH 


rH 


rH 


CM 


CM 


rH 


rH 


CM 


rH 


rH 


rH 


rH 


rH 




CM 




CM 


rH 


CM 


rH 







CO 






rH 


o 


rO 


H 


rfl 


H 


•H 


a 


u 


25 




p 


> 


Ph 


MH 


H 


O 








0) 








rfl 




25 


| 




MH 




O 




MH 


0) 




rH 




rO 




rfl 


rH 


*H 




M 




CO 




> 






ooooooooooooooooo 









M 


H 


















2: 
















U 


w 
















< 






2; 


















o 










CO 




CO 




M 






CO 




rJ 




H 




H 






CO 


CO 


HH* 




2; 




<J 






w 


CO 


(— 1 




P 










2: 


w 






Ph 




r^ 






rJ 


2: 


CO 








H 








W 




M 






O 






g 


> 




> 












w 


(— i 




(— 1 










o 


CO 




H 






W 


rH 






23 


M 


U 


CO 




o 


H 


w 




O 










M 


2: 


H 


o 


Ph 




P 




> 




w 


P 


CO 


CO 


U 


O 


1 


(— i 




H 


Ph 




w 


W 


Ph" 






W 


o 


3 




H 


Ph 




p 


CO 


Ph 








M-j 


M-J 


MH 


M-J 


4H 


4H 


4H 


o 


o 


O 


O 


o 


o 


O 


O 


O 


m 


MH 






m 


m 




m 


i ,' 



CM 
00 

o 

< 4J 

2S 

rH 

W 

H CU 



Cfl 

•H 
M 

> 

4H 
O 

0 

CO 



o 
u 



rHrHrHrHrHrHrHrHrHrHCMCMCMCMOvJCMCMCMCMCMCO 



232 



•H 

i— I t— I 

•H 0) 

4=1 > 

td a) 

4=1 >J 
o 

Ph 



»— I m 
o o 



o 



H 
O 



t— I t— I t— I t— I t— I H 

o o o o o o 
o o o o o o 



HHH H i— ! i— ! i— ! H H H H H H H H H 

ooomomv v oooovmoov oooooo 
oooooo oooo ooo oooooo 

pL|pL|...*pL|..*pL| PL, ..... . 

VVVAVAVVVVVVVA IJVVVVVVVVVVVVVVV 

PmPmPmPmPmPmHHPmPmPmPmHPmPmPmHPmPmPmPmP^PmH 

o o o o o 

o • o o 



-a 

=3 

c 

•H 
4J 

c 
o 
o 



w 
u 
53 

s 

Pn 





0) 






=3 






H 


m 




cd 


o 


a\ 


> 





t— I 



oincM<tovovoHvonooooHHNN^ONNcnvooocnnnriNfn 

W O ^ ^ O NNHHOOO>H^nNNinOn00OO>N0000HO><3-O 

I i i I I I I I I I I I I I I I I I I I I ( I I ! I I I I H 

i 



o 

o 

CO H 
^ H 
P O 

o w 

Pi CO 
CO O 
H 

53 

o 



o 



CO 

w 

H 



Pi 
W 
H 
H 
Pi 
O 



S3 
O 
H 
H 
<! 
o 

H 

H 
H 
CO 



CO pi 

13 P-i 



CO 
> 

P 

w pi w 



53 
O 



O 

CO W 

H Pn 

CO 

CO W 

H hh* 

CO ~ 

w 

H 



53 
o 

H 

SI 

P 
H 

> 

CO 
CO 

o 
Pi 
u 



O 













m 




m 


t— i 


m 
































H 
























o 




o 


o 


o 
































O 














4J 
































































•H 




1— 1 


H 


H 




H 








H 


H 


t— i 


1— 1 


1— 1 




1— 1 


1— 1 


1— 1 


H 


1— 1 


1— 1 


H 


i— 1 


i— 1 




1— I 


1— I 


»— I 


i— 1 


i— 1 


i— 1 


H 


H 


O 


O 


o 


V 


O 


V 


V 


V 


O 


O 


o 


o 


o 


m 


o 


o 


o 


o 


o 


o 


O 


o 


o 


V 


o 


o 


o 


O 


O 


O 


•H 


0) 


O 


O 


o 




o 








o 


O 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


O 


o 


o 




o 


o 


o 


o 


o 


o 


43 










Ph 




Ph 


Ph 


Ph 














































td 


5 






























































X) 




V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


A 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


o 




Ph 


Ph 


Ph 


H 


Ph 


H 


H 


H 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


1— 1 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 








o 




O 


o 


o 
































o 






























o 


































o 















0) 
4-> 
=3 
H 
O 

w 

< 



CD 
i— I 

td m 
> o 



CD 

td m 
> o 



43 
id 

•H 

td 
> 

o 

g 

53 



CCnC^CX)st^cOOONvDHNinNcCHCnC^NN<tCNvDOO<tH^ai 
mMTiH^nHCMM^^HNvO^vOrOMnNCnrnCOOOHHN^^H 



I— I 



mmooomoomoooooooomoinoootntninmmmo 
Hoo^^ONOroOvDNNnHairiiricoHOniriooiriiri^c^OooH 
o\voc^^o^roinro^H^HinoNinHvoc>Joo^ocn^oooc^o<^H 

i— ! i— ! NNNNCMrlrlHriNNrlrlCMrlrlHrirl CM CM CM rH 




oocooooooooooooooooooooooooo 



u-l m u-i HA ih *H 
]S £ & £ > > CO 



0) 

m i—} 

O 4=i 

td 

o u 
Z td 



HNn^invDNoo^OHNn^irivoNoo^Oricvjcn^invONooc^o 



233 



cd cy 

o 
}-< 

Ph 



o o o m m o o 
o o o o o o o 



V V V A A V V 
P-i ^ &4 ^ ^ ^ ^ 



w 
u 
S3 
w 



W CO 







fi-i Ph oo 




hh <rj cj> 




pi i i 








»2» o 




2 [5 !7 

W f*— 1 




PH L/ 




^ H 




HH l 1 




fj, Q 




w U W 








CO o 




~ , ^ 




CO £3 O 




pr") Q |_( 












•M ph 








> — 1 r~t HH 




b»: rv* i— t 




<tf O H 




CO 




CO Ph 


r— i 


r T 7 >— i 
HH m 


o 


I — ' hH 


CO 


HH HH PH 




5 p 5 


$ 






N CO ^ 




t> O 




Q 




w ^ w 




H W »_} 
















O CO 




CO W 




co ffl jz; 




< H O 




M 




ffi Ph H 




h o <: 




H Cn P 




12 H 




CO hJ 




CO w ^ 




H h3 > 




CO PQ 




W <j CO 




H H CO 








p % § 




> u 








W 1^ W 




Z Pfl 




H H 
















1 




i§ 





<3J +J 

H Uh 

cd o 
> 



4J 
•H 

iH H 

•H <U 

42 > 

CD CD 

pQ hh" 

O 



3 !3 

id 4-1 
> O 



4^ 

CO 
•H 
U 

cd 
> 

m 

o 

cu 

a 

CO 

S3 



CO 00 N CN O CM N 

o> oo co <r o \o <j\ 

<■ 00 00 H H O lo 

111 H H 

I I 



O O O LA LO O O 

o o o o o o o 



V V V A A V V 
Ph Ph Ph p., p-i Ph 



CN CO vo On o ro 
VO CM M H O ^ O 
fOiOHst OHH 

ON ON Q> H H O O 



lo lo O O O O LO 

oo ^ <i- i-H lo 

LO H ^ VO O CM Kt- 

co co <j- vo H H 

C J CN 



03 



03 
4J 



0) 0) 03 
4J 4J 4J 

43 tli Cfl 00 *H 
00 60 60 «H »H 
•H -H 'H 0) 0) 

0) o> o> 

& & & 

H CN 
n CN H I I 



03 

u 

o 



03 

e 

H 
X 



M-i M-l M-| LH M-l M_| M-| 

O O O O O O O 



H U 

11 

» S3 



H H H 

CU 0) cu 

111 

3 3 3 
S3 S S 



td rd cd rd cd rd rd 

4J 4J 4J 4J 4J 4J 4J 

O O O O O O O 

H H H H H H H 



ERIC 



CU 

m h 

O rO 
- «r( 

o u 
K cd 
> 



H N CO <}• m VO N 
VO VO vD VO VO VO 



234 



H H 
•H CD 
> 

ctf 0) 

O 
M 



o 



o o 



o 



o o o v o m 
o o o o o 
• • • p.! ♦ • 



H f— I f— I f— I H H H i— I H H tH rH rH rH 

vmoooommooLooo v voomo voo vo 
oooooooooooo oooo oo o 

VVVVII AVAVVVII A A II VAVvVVVVAVVVll VV 

o o o o o o 

o o o o 



o 
o 



<D 4J 

t— I M-l 

cd o 

> 



coNvo^cnoovoNcnooooHcninor^^incsico^ooNHvDNCNi^csic^ 
iririvON<rco^^voiriir)^so^coocoOH^cMcyirio^vooinincn 

I V I f I I I I C I I I I I I f | | | | | | | | | | | | | H 

I 



4J 



»H CD 
CD 

o 
u 



m h m 
o o o 



o 







tH 




tH 








rH 


rH 


rH 


rH 






tH 


tH 


O 


o 


O 


V 


O 


V 


V 


V 


O 


O 


O 


O 


m 


in 


O 


O 


O 


o 


O 




O 








O 


O 


O 


o 


o 


o 


O 


O 














pui 


Pk 


















V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


A 


A 


V 


V 


Ph 


Pk 


PM 


tH 


PM 


tH 


tH 


tH 


Pk 




Pk 


Pk 


Pk 


Pk 


Pk 


Pk 








O 




O 


O 


O 
















O 






O 





















m 
o 



o 



rH rH rH rH rH rH rH H H 

oooooomo voo 
oooooooo oo 



VVVVVVVAVVV V 



o 
o 



0) 
4J 

rH 3 N 

O H 

U) Cfl M-l 
JO > O 
<3 



voNO^ooHininrinoa\co<}-ocoN<}-ooNONoooooo<J-incncncNi 

C0VOCS|\O^NHNNCNl00 00Hfn<fiOHHOiriinO>^Hiri00a\OinCN| 



0) 
rH 

> O 



moommmioooooooomooommooinomoooino 

inN<i"ro^HHOHinoincnnHOH<fchH<forN<}-H^ri(0(nco 
^<f^vor^oo<fCj>a\HcsivDininroo<f<fcnoin^cNj\ooo^rHrHOvo 
coco^cnH^^iooNO^HOinin^ocnHvoocsi^c^vococniricNi^cNi 

HHHCSlCvJCSlCvlCSJHHHCNlCNJNHCNlNCNlHCNlCNH CM CM H CM N 



•H 

> 

tH 
O 

<D 

1 




q-i q-i q_| q-i 
O O O O O 



M-l M-l M-l M-l t|k M-l M-l 

o o o o o o o 



ERLC 



Q) 

M-l rH 
O ,£> 

O M 
525 cd 
> 



HcMcn^^lovo^coo^oHc^lcn>^lnvo^ooc^OH^lcn^^lnvo^ooo^o 

HHHHHHHHHHCMCMCMCMCMCMCMCMNCMr) 



235 



o 



o 

CO 
H 
CO 

w 

H 
4J 



O 



Ptf CO 
W H 
Pn 
Pn 



CO 



o 
u 

CO 
Ph 

8 



O 
H 
H 
_ O 
Pi W 
O CO 



53 
O 
f— I 



•H 

rH rH 

•H 0) 

4=1 > 

Cd <D 

o 
u 

Ph 



Q) +J 

i— ( M-l 

CO O 
> 



U 
•H 

i— I i— I 
•H CD 

> 

cd cd 

O 

Ph 









m 






LO 










tH 






tH 


rH 








rH 


rH 






m 




in 




rH 


m 












o 






o 










O 






o 


O 








O 


O 






o 




o 




o 


o 


































































CD 


H 


i— ! 


H 




tH 








i— I 


rH 


tH 














rH 


rH 






tH 


tH 




tH 




tH 






tH 




O 


O 


o 


V 


o 




V 


m 


o 


O 


O 


V 


m 


m 


V 


V 


m 


O 


O 


V 


V 


O 


O 


V 


O 


V 


O 


V 


V 


O 


a 


O 


o 


o 




o 


o 




o 


o 


o 


o 




o 


o 






o 


o 


o 






O 


O 




o 




o 






o 


*H 








Ph 






Ph 










Ph 






Ph 


Ph 








Ph 


Ph 






Ph 




Ph 




Ph 


Ph 




4J 

C 


v 


V 


V 


V 


11 


A 


V 


A 


V 


V 


V 


V 


A 


A 


V 


V 


A 


v 


v 


V 


V 


V 


V 


V 


v 


v 


v 


V 


v 


v 


o 
o 


Ph 


Ph 


Ph 


I— 1 


Ph 


Ph 


tH 


Ph 


Ph 


Ph 


Ph 


tH 


p-i 


Ph 


tH 


rH 


Ph 


Ph 


Ph 


rH 


rH 


Ph 


Ph 


tH 


p-i 


tH 


Ph 


tH 


tH 


Ph 










o 






o 










O 






o 


o 






o 


o 




o 




o 




o 


o 




























O 






o 


o 








o 


o 














o 








CO 


m 


CO 


00 


o> 




CM 


CO 


rH 


CM 


tH 




CO 


CO 


00 


tH 


m 






m 


tH 


O 




O 


a> 


CM 




CO 


-H 


O 




00 


CM 




CO 


CO 


00 


CM 


00 


rH 




VD 


O 


m 


CO 


O 


CM 




00 


CM 


tH 


CT\ 






O 


m 


m 


o 


CM 


CO 






i 


<r 
I 


m .cm 
I I 


CO 

1 


t— 1 
1 


CM 
I 


I— 1 

1 


i 


CO 

I 


VD 
1 


CO 
1 


rH 
1 


H 

1 


CO 

1 


CO 

1 


tH 

I 


co 
I 


<r 
I 


CO 
1 


CM 
1 


I 


I 


CM 
1 


1 


CM 

1 


1 


CO 

1 


CM 
1 


tH 
tH 
I 










I— 1 




m 


tH 


m 


















m 














m 




m 






m 












o 




o 


O 


o 


















o 














o 




o 






o 






i— i 


i— l 


i— I 




rH 








tH 


tH 


tH 


tH 






tH 


tH 




tH 


tH 


tH 


tH 


tH 


tH 




tH 




tH 


tH 




tH 




o 


o 


o 


V 


o 


V 


V 


V 


O 


O 


O 


O 


m 


m 


O 


O 


V 


O 


O 


O 


O 


O 


O 


V 


O 


V 


O 


O 


V 


O 




o 


o 


o 




o 








O 


O 


O 


O 


o 


o 


O 


O 




O 


O 


O 


O 


O 


O 




O 




O 


O 




O 










Ph 




Ph 


Ph 


Ph 










































Ph 






V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


A 


A 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 1 





Ph Ph Ph tH Ph tH tH tH Ph Ph Ph Ph P-j Ph Ph Ph t — I Ph Ph Ph Ph Ph Ph t — I Ph t — I Ph Ph t — I 

o o o o o o o 

• • ■ • ■ • 



o 
o 



Ph 



CO 

w 
> 



PS P4 

O H 
H 
W CO 



p 
w 

H 



S3 

w 

PS 
H 

O 

w 
rJ 



o 



a 
o 
co W 
CO B 
<! H 



S3 

o 

H 
H 
<U 
P 

£ H 
CO " 
CO W 



o 
p^ 



co pq 
w- < 

H H 
> 



! 

g o 



3 

> 

CO 
CO 

o 
PS 
a 



a) 

•P 

rH 3 N 
O H 

^ > O 



a) 

rH 

> o 



o 


CO 


IT| 


CM 




m 




tH 


o 


O 




CM 


CO 


CO 


tH 


tH 


CM 




rH 


rH 




o 




o 




CM 


CO 




tH 


CM 




o 


<T 


CM 


m 




ON 


CM 


CO 


CM 


G\ 




o 


o 


tH 




00 


o 




CM 


rH 




CO 


o 


o 






m 








CM 


O 


KD 


m 


CO 


m 


O 


<r 


CO 




m 






00 




<r 


o 


CM 


rH 


CO 


00 




o 










<r 






m 


o 


CM 


CO 


CM 


CM 


CM 






KD 


CO 


rH 


tH 


CO 


CO 


CM 


<r 


m 


<r 


CO 






CM 




CM 


m 


CO 


CM 




m 


o 


o 


m 


m 


m 


m 


O 


o 


m 


O 


m 


m 


m 


m 


O 


m 


o 


o 


o 


o 


o 


o 


m 


o 


m 


in 


m 


o 


m 


o> 


o> 






m 


tH 


CO 


o 




CO 


m 


o 


CO 


m 


m 


CM 


m 


m 


<r 




a> 


tH 


o 


tH 




o 


o 




CM 






CO 


00 




r^s 


00 




a> 






tH 


00 


m 


m 


m 




<r 




<D 


c 


m 


CM 


tH 


a> 


m 


o 


a> 


CM 


O 


m 


00 


00 


CO 


CO 


rH 


o- 




m 


o\ 


00 


tH 


o 


m 


m 


OS 


o 


CO 


tH 


vD 


o 


CM 






m 




<r 




CM 




CM 


tH 


tH 


tH 


CM 


CM 


CM 


CM 


CM 


tH 


tH 


tH 


CM 


CM 


CM 


rH 


CM 


CM 


CM 


t—1 


CM 


CM 


tH 




CM 




CM 


tH 


CM 


CM 






o 

H 

o 

H 
H 
O 



H 



W 
Ph 
W 
P 



H H 

PS » 

O PJ 

< S 



oooooooooooo 



^^^^^^^^^^ 



o o 















CO 












CO 














1 


CO 


CO 


d 








M 




w 


CO 


H 
















^S 
















CO 






P 








w 






> 


H 


O 


H 




> 




> 


H 


O 


M 


>J 


w 


H 




H 


H 


W 


CO 


M 


a 


CO 


o 


H 


?i 




CO 




PS 




H 


U 






w 






o 




P3 


H 


W 


p^ 




o 


Ph 


g 


P 


HH 




o 




CO 


CO 


O 


O 






PS 


W 


w 




w 




H 


H 


Ph 


PS 


PS 




H 


Ph 


m 


m 


UH 


m 


m 


m 


MH 


«H 


o 


o 


O 


o 


o 


o 


O 


O 


m 








<H 


m 


m 


<H 




18 


18 


18 











55 

o 

H 
H 

H 
H 
O 

w 
o 



m 

bo 
o 

4J 



W CO 
PS 

to 

CD 



> > 

P CO 

m m 

o o 



H PJ 
S3 H 
W p 
H Ph 
O W 
PM B 

m m 
o o 



o 

H 

H 
W 
CO 
CO 

< 

o 

4H 



03 

-H 

id 
> 

MH 
O 

CO 



0) 

m H 

O 

cd 

■ »H 

o u 
& cd 
> 



Hcvin^in^Nco^OHNcn<riovDN(X)^OHcsiro<rinvDNOOo>o 



236 



ERIC 



4-> 

•H 

H H 

•H 0) 

> 

0) 

o 



o o o to to o o 
o o o o o o o 



V V V A A V V 
P-i Ph Ph Ph Ph Ph Ph 



0) 






m 




o o 


00 


CN 






co 


o> 


00 


o o 




H 


H 


m 








• • 








o 


00 


on 


00 


O H 


00 


CO 


> 




1 


I 


1 


1 


I 


iH 



•H Q) 

> 
Q) 

O 
U 



o o o in in o o 
o o o o o o o 



V V V A A V V 

PL, PL, P-| PL, pL, P-| PL, 



<U 

3 0) 

O iH 

co tti M-r 

•9 > ° 
< 



O sr ro vo O ro <J- 

CO VO N VO O M is 

o h vo io o co <r 

oo on oo o h on on 



0) 

m-i 
> o 



in ^ o o o in in 

oo h in (N h in N 

ON O vO O O H O 

vo <t io n n <N m 



0) 
CO 

0) CO 
CO CO H & 



CO CO CO 4-> 4-> 

4-) 4-) 4-) & & 

^ ^ ^ 60 W) 

00 00 00 -H «H 

rl rl *H Q) QJ 01 X 

0) CU 0) £2 0) 

£ £ ^ T3 

H CM O Pi 

CO CN H I I H 



M-l 


M-l 


M-l 


M-l 


m-i 


M-l 


m-i 


O 


O 


O 


O 


o 


O 


o 
















<u 


CD 


CD 


0) 




CU 


cu 
















■§ 










"I 


1 






















S3 








al 


al 


al 


al 


al 


al 


al 


4-) 


4J 


4-) 


4-) 


4-) 


4-) 


4J 


O 


O 


O 


O 


O 


O 


O 


H 


H 


H 


H 


H 


H 


H 



CU 

M-l H 

0l S 

• -r! 
O M 
S id 
> 



H CN ro ^ iO vO N 
vO vO vO vO vO vO vO 



237 



w 
u 

2; 2; 

w o 

Pi CO 

W M 
pi 



o 

5 ° 

2 CO 

o 



Ph 
o 

CO 
H 
CO 

w 

H 

-U 
Q 



U M 



CO W 
W P-i 

p-l 
p 



> 



Q 
W 
H 
< 
M 
U 
O 
CO 
CO 

< 

n 

H 

55 



CO 

> 
w 

g 



CO 
CO W 

CO PQ 

w < 



P 



2S 
O 
H 

^ 

P 
H 

> 

CO 
CO 

o 

Pi 
u 



O 



4J 



•H 0) 

cd a) 

O 
Ph 



o 



LP) 

o 







0) 


4J 














H 


H | 






cd 


o 


CO 




> 




o> 








H 
















21 








o 




4J 




H 




«H 




H 




t— 1 


t— 1 


U 




•H 


0) 


W 




rQ 


> 


CO 




cd 


0) 










2; 




o 




o 








H 




Ph 




H 








< 








U 








M 
















M 


0) 






H 


4J 






CO 




0) 






rH 


S3 


N 




O 


iH 






CO 


cd 






• rQ > 


o 


H 


< 






23 








O 




0) 










P 


W 




i— i 








cd 


M-J 






> 


O 



cd 

id 
> 

CM 
O 

0) 

0 
td 

525 



LP) 

O 



LP) 

O 



oooooooo 



m m m 
o o o 



V LP) LP) V LP) V 

o o o 



LP) LP) 

o o 



O O LP) LP) LP) 

o o o o o 



V LP) I — I O 

o o o 

PL, . . • 



AAAAAAAAVAAAVAAVAVAA V II II A A A V A II 



o 
o 



o 
o 



o r^. on o t— i t— i 



i i i 



I I I i i I I I i I i i i i 



1— I LP) 

o o 



UO LP) 

o o 



LP)LP>LP)LP)LP)LP)LP)LOV V LP) LP) 

OOOOOOOO oo 

PL, PL, . . 



LP) LP) LP) LP) V LP) LP) 

o o o o o o 



AAAAAAAAVVAAVAAAAVAA 



O O 
O • 



1— 1 


p-l 


p-l 


p-l 


P-i 


P-i 


t— 1 


p-l 


P-i 


p-l 


o 












o 

• 










o 




CO 


i— 1 


LP) 


00 


00 


CsJ 


oo 


i— 1 




si- 




<D 


VO 


CO 


00 


v£> 


CsJ 


CM 
1 


oo 

1 


oo 
I 


H 
1 


H 
1 


H 
1 


CsJ 
■ I 


o 
1 


CM 
I 


CO 

1 


LP> 


1— 1 










LP) 




LP> 


1— 1 


o 


o 










o 




o 


o 


























i— l 
















V 


V 


o 


LP> 


LP) 


to 


V 


LP) 


V 


V 






o 


o 


o 


o 




o 






P-l 


p-l 










p-l 




p-l 


p-l 


V 


V 


V 


A 


A 


A 


V 


A 


V 


V 


1— 1 


mi 






P-. 


P-. 


1— 1 


P-. 


1— 1 


1— 1 


o 


o 






o 




o 


o 




o 
















o 



Oooo<rHcs]vocn^in^c\]cncvjcvi^LnnNONcnvonoowcsiNcvirnvo 
oo oo cm ct» cm nN^ooric^oo^votnin^ooo^Ocn^HnN^inoo 
oON^^ONno^Hcsi^^cvjcviooonvooNinHinrornooco^^ON 

OHHOOOOOCNlcMOOcMOOHHMHOcMnnHHHMOCMM 



LO LO LO LO CO LO LO <0 LO LO MO LO LO LO LO LO LO LO 

HvOCNjLnstn^HriOrsloOONONrsJNcO^I^I^^CONOvt^O^H 

noOintonHOvonvococvioooHn^cvicsiHvonoo^ooininvooo 
NLon^vONvoNOcs]oincnN^nLr)cvi(nLOcvio>ooLn(nnHincNicr> 

CMCMCMCMCMCMCMCMCMCMCMCMCMCMCMCMCMCMCMCMC^ 



o 
23 



s 

Pi 
p-l 




523 



Pi 
o 
w 



w 
Pi 

(— I 
H 

S2 



O 

2: 

M 
25 
O 
M 



M 
H 

5 



W 
P-i 
W 
P 



H 

2: 
w 



> 
w 

u 



E-4 

2; 



CO 



o o o o o 



o 


















H 




25 










1 


i 




H 


o 














5. 




H 


W 


M 




> 


H 


o 


U 






U 


»J 


o 


H 


U 


M 


M 


H 


a 




M 


2; 




W 


CO 


25 


U 






PQ 


M 


3 


>J 


CO 




P 


w 




H S 




>J 


w 


n 




Ph 




X 


o 


H 


W 


Pm 






o 




w 


o 


M 




o 




o 


c 




hJ 


Pi 


25 




pi 


U 


u 


u 






o 


H 


H 


Ph 


MH 


MH 


MH 


MH 




M-l 


M-l 


M-l 


M-i 


O 


O 


O 


O 


o 


o 


O 


O 


O 


MH 


MH 


MH 


MH 




M-l 


M-l 


M-l 


M-l 



CO 

CO CO 
W CO 

2; W ^ 
tJ 2 w 

w 
> 

,J H • ■ 
M U CO U 
PQ Ph 2 H 

3 P o g 

h o p., a 

CO CO 

§§§ 



2; 

s 



H 
U 
P 
P 
U O 
PJ Pi 
H Ph 



CO 



2: 
o 
f— i 

S3 

> 
(— I 
H 
O 

W 

u 



> > 

h pi 

pi Pa 

P CO 



2: 

w 
W 
H 

o 

H 



0> 

Cv) 

CxO 

o 



w 

H 

p . . 
P-i CO > 



0} 
0) 
I— I 

td 
td 



M-l M-l M-l M-l 

o o o o 



M-l M-l M-l 
O O O 



O W CO 

Ph P5 < 

M-l M-l M-l 

O O O 



M-l 
O 



-a 

§ 

4J 
O 

u 



M-l i— I 

O rO 

cd 

o u 
2: cd 



HcNicn^invONooc^OHcsicn<rinvONooc^OHcN(n<rinvONcoo>o 

HHHHHHHHrlrlCs|Cs|CslCVlNCslCNCs|CslCs|rO 



238 



w 
o 

S3 S3 
w o 

Ptf co 
W M 

£^ 
°s 

5 ° 
w eo 

o 

o 

CO 

H S3 

CO o 

W H 

w 

+J H 
M 



CO W 



> 



> 



s 

H 
<J 
M 
O 
O 
CO W 

3g 



W 
P 

9 



H 
H 



CO 
CO W 

co pq 
w < 
H 



S5 
O 

W 



CO 

o 

H 

H 

M 



2; vo 

H 
H 



ro 
to 
C 
Pi 
O 

w 



h3 
< o 



S3 
o 



4J 



•H 0) 

o 



o 



o 



ooooooooo 



in m m m ^ iri in 
o o o o o o o 



vinminoominm 
oooooooo 

PM 



m 
o 



v m v o 
o o 



AAAAAAAAIIVAAAAAAAVAAAVVAAAVAVV 

o o o o 

o 







0) 




o 


CO 


CO 


CM 


CO 


i— I 


m 


o 






i— I 


CO 


CM 












o 


CM 


VO 


CM 


00 


m 


CM 


o 


CO 


CT> 






00 


CO 






i— I 
































/-\ 




id 


o 


o 


i— 1 


i— 1 


1— I 


o 


o 


tH 


o 


CO 


i— 1 


O 


O 


i— 1 


O 


CO 




> 




1 


I 


I 




1 








1 


1 


1 


I 


1 


1 


o> 




































I— 1 




















































































m 












i— ! 


m 






m 




ft 












o 












O 


o 






o 




o 




4J 
































H 




•H 
































H 




H 


tH 


m 




V 


m 


m 


m 


m 


m 


V 


V 


m 


m' 


V 


m 


o 




•H 


0) 


o 


o 




o 


o 


o 


o 


o 






o 


o 




o 


w 




•s 


& 






























CO 












































A 


A 


V 


A 


A 


A 


A 


A 


V 


V 


A 


A 


V 


A 


o 




o 
u 




Ph 


P-i 


i— ! 




PM 


PM 


P-I 




iH 


H 


P^ 




i— I 


P-i 


H 




P-i 








o 












O 


o 






o 




H 






















O 












<J 




































O 




































H 








































































H 


(1) 


































H 


4J 


































CO 




0) 




i— I 


i— I 






O 


a> 


CM 


<r 


00 




CO 


<r 


VO 


h 




tH 


S3 


N 


VO 


m 


CO 


o 


o> 


CO 


tH 


CnJ 




vD 


00 


iH 


H 


VO 




O 


H 




CO 


o 


o 


VO 


m 


o 


00 


CO 


o 


iH 


CO 




CO 


CM 




TO 


cd 


m 






























W 


tS 


> 


o 


o 


i— l 


CM 


o 


o 


o 


O 


o 


CO 


CM 


o 


o 


CM 


O 


w 





































m 
o 



m 
o 



m 
o 



m 
o 



P-i 



m 
o 



v m m v 
o o 



P-i 



oommmvinvo 
o o c o o o o 
P-i •P-i • 



AVAVAAVVVAAAVAVV 



o 
o 



HNOONOOmHmCN 



N O CM O H H 
O CT* CM CT> 00 CM 



cn oo o r>» m 

O H vO <T CO CM 

o cn co fN vo o> 



r>s o 

H CM 
<J" CO 



OCMOCMiHOCMCO^tiHiHiHCMOCMcO 



O m 



0) 




m 


m 


O 


m 


O 


in 


m 


O 


in 


O 


m 


m 


o 


O 


O 


m 


m 


o 


m 


O 


o 


m 


O 


m 


m 


O 


O 


m 






00 


CM 


00 


vo 


O 


CM 




CO 


CO 


m 


CT» 


o 








tH 


00 






00 


CO 


CO 


CM 


00 






VD 


00 


cd 




m 


o 




CO 


CM 


CO 


o 


o 


CM 


tH 


CO 


00 




o 


CM 


VO 


m 


m 


VO 


m 




On 


O 


00 


m 


o 




m 


> 


o 


VO 


m 


CM 


VO 


VO 




vO 




o 


CM 


vO 


m 


co 






CM 


m 


CM 


CM 


m 


CM 






m 


CM 




o 


m 






CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


H 


H 


CM 


CM 


CM 


CM 


CM 



0) 
tH 

*n 

cd 
H 
U 
rd 
> 

O 
0) 

1 




iu m \u w-i 
S £ > ? 



QJ 

g 

•H 

c 
o 
o 



m tH 
o *n 
cd 

O U 



HCNC0<tinvOI^COO\OHCMc0<3-invOr^00CJNOiHCM 

cococococococococo^t^tsj-sj-sj'sj'^j'^J'sj-sj-ininin 



co<j-tnvor^ooo\o 
mmmmmmLnvo 



239 



u 







m 


H 








H 




>> 




o 


O 








O 




u 


















•rl 
















1— 1 


H 


H 


V 


V 


m 


m 


m 


V 


o 


■rl 


CD 






o 


o 


o 




o 






PM 


PM 








PM 




cd 


£> 




















V 


V 


A 


A 


A 


V 


II 


o 






















H 


H 


PM 


PM 


PM 


H 


Pm 


PM 




O 


O 
O 








O 

o 

































O 






0) 








o 


o 


o 


m 


m 


S 


CO 










LP) 


o 




o 


o 




CM 


W 


M 






rH 


















hH 


cd 






Cd 


o 


CM 






rH 


rH 


CM 


CO 




< 


CO 








| 


I 


| 






| 


J 


f— ( 




0> 






















p 




f— H 
























Q 


























CJ 












rH 








H 


H 














O 








O 


O 




CO 


o 




4-) 




















PM 


H 




■H 




H 
















p 


H 




H 


H 


O 


V 


m 


m 


m 


V 


V 


Pm 


o 


U 




•rl 


CD 


O 




O 


o 


o 






o 




W 






fc 




PM 








PM 


PM 




a 


CO 




cd 








^ 










co 








►J. 


V 


V 


A 


A 


A 


V 


V 


H 


ss 


53 




0 




















o 


o 








Pm 


H 


Pm 


PM 


PM 


H 


H 


w 


H 


hH 




Pm 






O 








O 


O 


H 




H 








O 








O 


O 




0 


<J 






















4J 


H 


u 
























H 


H 


























Pm 
























U 


H 


CD 
























H 


4J 






















ps! 


CO 


.1) 


cu 




CM 


<r 


<r 


O 


O 


CTn 




CO 


Cd 


p 


rH 




N 


CM 


o 


CM 


o 


o 




o 


W 


PM 




0 


H 


m 


o 




o 


o 


00 


o 


P 


PM 


W 


CO 


cd 




















P 




> 


O 


ro 


ro 


H 


H 


o 


CM 


ro 








< 




















> 




H 
























CO 
























N 


> 


55 
























O 




CD 




O 


o 


O 


O 


o 


O 


o 




W 








P 
















a 




w 




H 




m 




vO 


H 


00 


O 






Q 










CM 


m 


H 


O 


ro 




m 










> 


0 


00 




CO 






a> 
















i— 1 


t— i 


CM 


CM 


CM 


i— I 


t— i 

































CD 
















CO 














4J 


P 














X 
















CD 


CO 










CO 


CO 


H 






CO 


CO 


CO 


u 


4J 




g 


CD 


4J 


+J 


4J 






a 


CD 


H 








00 


00 


•H 


H 




00 


00 


00 


•rl 


■rl 






•§ 


•rl 


•rH 


•H 


CD 


CD 


CO 




■rl 


CD 


CD 


CD 


& 


& 




9 


JM 


& 




& 






JM 




id 








H 


CM 


0 




> 




CM 


H 


1 


1 


& 




MM 






M-J 


m 


M-J 


MH 




O 


O 


0 


0 


0 


0 


0 


0 


0) 


>M 




Jm 


Jm 






JM 


a 


CD 


CD 


CD 


CD 


CD 


CD 


CD 


















& 






1 




•§ 








1 












1 










53 


i 


53 






al 


al 


al 


al 


al 


al 


al 




4J 


+j 


4J 


4J 


4J 


4J 


4J 




0 


o 


0 


0 


O 


O 


0 




H 


H 


H 


H 


H 


H 


H 



CD 

M-l rH 
O H^ 

cd 

. -rl 

o u 
55 cd 
> 



h cm in ^ m vo n 

vO vO vO vO vO vO vO 



ERIC 



240 



u 
•H 

H H 
•H Q) 

rQ > 

a) 

o 

M 



o 



ooooooooo o.o oooooooooooo 



in in tri in in in 
o o o o o o 



AAAAAAAAAAAAAAAAAAAAAAAVAAAAAA 

o 



c 



c 
o 
o 



o 

2 o 

S3 

2 O 

3uh 



CO 

D o 

O H 



o 



CO O 
H 



CO 

w 

H 



S3 
O 
H 



5 psS 

co p*3 

w w 
> 



N 



o 
W 

CO 

S3 
O 
f— I 

51 



W 

rd 

H 
O 



CO 

> 

PA W 



CO 



a 
o 

co w 
co JU 55 

< H O 



H 
CO 

W 
H 



Pi 




o 


51 




N 




H 


CO 




w 










W 




O 


> 


w 














►J 


o 


S3 




o 





Q) -P 
rH M-l 

o 

> 



•H 

rH rH 

•H CD 

,0 > 

O 
M 
P-i 



0) 

■u 

O iH 

CO if : 
^ > O 



0) 

mh 
> o 



00CNlcnNNH00Oin^00CvlNHOH^vDC0'no>NvDcncs|vDCv|^H^ 

OOOOOHOOOHOHHHOHHHHOOOOCMOOOr-IHO 
! I II I III I I I I I ! I I I I 



m m 
o o 
• • 

inminmininminin v minininininininminminio 
ooooooooo ooooooooooooo 



m m to m m m 
o o o o o o 



AAAAAAAAAVAAAAAAAAAAAAA 



A A A A A A 



o o 



CM 


m 


m 


o 


0\ 


CO 


in 


o 


o 


vO 


o 




st 


O 


00 


CO 


m 






CO 


CO 




CO 


m 


CO 


rH 


CO 


m 


CO 




CT> 


st 


o 


CO 


st 


m 


m 


o 


o 


<D 


o 


m 


st 


O 


CM 




o 


o 


st 




O 




CO 


st 


o\ 


CM 


m 




st 


m 




CO 




o 


<o 


H 


m 


o 


CM 


G\ 


rH 






CO 


o 


a\ 


m 


rH 


CO 




vO 




vO 


m 


CM 


m 


st 


st 


o\ 


rH 


o 


H 


o 


o 


o 


i— ! 


o 


o 


o 


t— ! 


o 


o 


rH 


rH 


o 


o 


i— I 


i— I 


i— ! 


o 


rH 


o 


o 


CM 


o 


o 


O 


rH 


o 


o 


o 


m 


o 


m 


o 


O 


o 


o 


m 


O 


m 


o 


O 


m 


m 


o 


m 


m 


O 


o 


o 


m 


m 


o 


o 


o 


O 


O 


o 


m 


st 


o 


m 


CO 


1^ 






CO 


o\ 


o 


st 


CM 


1^. 


o 


CO 


CO 


m 


st 


m 


CO 




o 


CO 


o 




H 


CO 


o 


o 




00 


CO 


CT> 


CO 


H 


CM 


m 


CO 


a> 


m 


rH 


o 


CM 


o> 


CO 


CO 


rH 


CM 




CM 


CM 


m 




o 




vO 


st 


CO 


CO 




m 


st 


m 


r^. 








r^. 


<D 


co 






vD 


m 




m 


m 


m 


st 


<D 


m 


m 


m 


CO 


vD 






st 


m 




CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 




14_|14-|14_|14_|14_|14_|14_|14_|14_| 



m m 4-i 4-i m M-i 4-i 



m 
o 



Hcvico<fLn^Nco^OHcNin<fLn^NCOo>OHcNin^Ln^rNooo>o 

HHHHHHHHHrlMCMCMCMCMCMCMNCMCMcn 



241 



•H 0) 

rO > 
Cti (1) 

o 

U 

Ph 



m 
o 



o 



inininintnini^inininmininininininv in m m m ^ v m m m m m m 
ooooooooooooooooo ooooo oooooo 

• • • Ph Ph 



0) 
•H 



AAAAAAAAAAAAAAAAAVAAAAAVAAAAAA O 

O 

o o 



w 
u 

w 



£3 
> 



55 
O 



W CO 

Fn pi 

M < 
P 



O CTs 

O H 



S CO 
Ph 
J*h p 

o o 

pi 

co o 



CO 

w o 

H H 

Pi O 
■U W M 

H H 
g M 

< O 

CO Pi 



< 

3 



Ph 

Ph W 
H 



N CO 
> 



£5 
O 



< 
M 
O 
O 

CO W 



Pi w 



o 



CO 

o 

H 
H 

- <t| 

M 



Pi 
o 



H 

IS 

CO 
CO W 

CO PP 

w 

H 



> W 

Fn 
h4 O 



S3 
O 



o 

> 



•H CD 

> 

a) 

O 

u 

Ph 



0J 
4J 

2 .0) 
H 3 N 
O iH 

rQ > o 



0) 

3 S3 
i— i 

> o 



inHOO(o^^oovocoincNj^ocovo<rinNnincNjcT»riONo>^vo 

HOHOOOOOHOrHr-|r-IOOr-|r-|CVlr-|OOr-|OCNIr-IOOr-|r-|iH 

I I I I I I I I I I I I I I I I I I I III 

I— I 

o 

oooooo ooooooooooooooooo oooooo 



A A A A 



AAAAAAAAAAAAAAAAAAVAAAAAA 



PmPhPhPhPhPhPhPhPhPhPhPhPh^PmPM^^^^^^^^PliP^P^P^PhPl, 

o 



ir^r^r^cvji— i o^coc^^^^csir^r^^Dcs]LnLn<tcs]Ln<tOr^csir^o> 
Hc^oon<i-ino<rN^<r^ocNjvono>Ln<rcNn<tHHroc^ovooo 
^^lo^oolnHlno^Orlrl^oo( f llnnc^J^vo<r^o^oolnc^]voo^ 

rHOrHOOrHOOOiHiHrHiHOOrHiHiHiHOiHiHOCNJOOOiHiHiH 



oooointninotntnoootntnininoioinooinooinooino 

OOONM30^con<t^ i— ir^r^<tr^Or^cs]Oma><t^DOa>Lnr^a>oo 
OLnOroroojLncoa^csi^D co cn co co t— i h n cm n <r m n m tn o> 

tnin^fNVDvovofNinin^in^fxNinin^^^oincn^ocNinvovo^incN 

CS1CS1CS1C\1C\1C\1C\1C\1C\1C>JCnJCnJC>JC\ICSIC^ 




m iw iw m m m 

s £ s s s s 



mmmq_|ij_|i4_|i4-|i4_|i4_|i4_|i4_|i4_|i4_| 



i^H q-l M_l M_l q_l M-l 
£ £ £ £ £ £ CO 



m h 

O rO 

• «H 

O >-l 
525 id 
> 



ERIC 



242 



0 

ERIC 



•H 

H H 

•H 0) 

4=1 > 

cd a) 

43 >J 

O 
PM 



in m o m m m 
o o o o o o 



V A A A 



V 
PM 

V A A 



H P-i P-i PM P-i PM P-i 

O 

O 



cd o 



<t cn cn cn o oo <t 
oo cn n o> O rl o 

CM O O CO «H O O 



4J 

H H 

•H 0) 

43 > 

td Q) 
o 

PM 



v m m o m m in 

o o o o o o 

PM 

V A A V A A A 

H PM PM A* PM pM 

O 

O 



0) 
4J 

3 0) 
H 3 N 
O H 

tS > ° 
<c 



(N O <f 00 O (Nl <t 

OMH ^) O N H 

o vo <t* o m «— i 

CO O H sf H O O 



CO 



0) 

3 S3 
H 

td m 
> o 



m o m in o o m 

vO O ^ ^ H 0> 00 

^ oo m in o oo o 

o\ m m o\ in r- 

H Cv| CM H CM N M 









03 


0] 


to 


OT 


CO 


4J 


•u 




U 


4J 








»tS 




W) 


60 


00 


W) 




•H 


•H 


•H 


•H 


0) 


0) 


0) 




0) 


S5 


ts 


















H 


CM 


CO 


CM 


i— 1 


1 


1 


m 


m 


m 


m 


m 


o 


o 


o 


o 


o 












0) 


0) 


0) 


0) 






43 


43 




*2 



0) 
CO 



0) 
H 

0) 

M 





53 




53 


53 


53 


H 


iH 


i— 1 


H 


i— 1 


H 


cd 


cd 


cd 


cd 


cd 


cd 


4J 


4J 


4J 


4J 


4J 


4J 


O 


o 


o 


O 


O 


o 


H 


H 


H 


H 


H 


H 



0 o 
u u 

0) 0) 
43 A 

1 l 

53 



4J 
O 



0) 

m H 

O 43 

• t4 
O M 

525 cd 
> 



H N CO^ m vO N 
vO vO vO vO vO vO vD 



\ 



243 



•H 

i— I i— I 

•H 0) 

4=1 > 

a) 

o 
u 

Pn 



m 
o 



lo 
o 



• • • 

=3 

v lo lo lo lo lo lo lo lo v ininminiAiAinininmin^ininininv lo lo lo c 
oooooooo OOOOOOOOOOOOOOOO OOO *H 



Pn 



VAAAAAAAAVAAAAAAAAAAAAAAAAVAAA 



C 

o 
u 



o o o 

o 



pi co 
w 

P^ 



5 



£ Pi 

o o> 

U tH 



o 

CO 
H 
CO 

w 

H 

4J 



CO 

w 
> 



CO 

Pn 
p 
o . 

o u 



53 

o 



N) CO 
> 

P 

w pi 

<j s 

H O 

U >J 
O 

CO W 

H PL4 

E2 

CO 
CO W 

H >J 

CO 

w 

H 



W 

CO 

o 

H 
H 
<J 

6 

w 

H 

S3 
O 

W 



CO 
S3 

o 

M 



O 
W 



a vo 

H O 



O 



0) 




o 


H 


m 


o 


00 


<r 


rH 


I— 1 


o 


o 


vo 


00 


o 


LO 




LO 


o 


CO 


o 




o 




00 




o 




CO 






o 


=3 




<• 


i— 1 


H 


a> 


i— ! 






o 






o 


T— 1 


o 


CM 




00 


o 


00 


o 




o 




CO 


00 


o 


CO 


LO 


H 




00 


i— 1 


m 






























































Va 


o 


CM 

I 


o 


o 
1 


o 


H 


rH 


o 
1 


I— 1 

1 


o 


CM 


rH 


o 
1 


rH 

1 


T— 1 


o 


o 
1 


o 


o 
1 


o 
I 


o 


I— 1 

1 


o 
t 


H 


o 
t 


o 


H 


CM 


O 
t 


o 


o 



4-) 
•H 

i— I t— I 

•H CD 

rQ > 

a) 

4=i >J 
O 

P^ 



0) 
4-J 



LO 

o 



LO 

o 



V LOLOLOLOLOLOLOLOV loiOlolOLOlOlOlOlOiOLOiOlOlOlOlO V LO lo lo 
OOOOOOOO OOOOOOOOOOOOOOOO OOO 
Ph Ph pui . . • 

VAAAAAAAAVAAAAAAAAAAAAAAAAVAAA 

rHpHpHpHpHpHpHpHpHiHpHpHpHpnPMPnP^pMpMpMpMpMpMpMp^ 

O O O 



3 


0) 




CO 




LO 




CO 


rH 


rH 


O 


00 


O 


rH 


00 


O 


o 






o 


O 


LO 


00 






00 


00 


ON. 


vO 


CO 


00 


LO 


LO 


iH 


=3 


N 


CO 




rH 




ON. 






rH 






00 


CO 


o 




CM 


00 


o 




CM 


<r 


CO 




LO 


oo 


<r 


rH 


CM 


CM 




O 


O 


rH 


rH 




O 


LO 


CO 




CM 


o 


CM 


a> 


LO 


O 


o 


a\ 




O 


O 




CM 


LO 




rH 


<r 


ON 


O 


CM 


LO 


CM 


CM 


00 


03 


cd 


m 






























































Ab 


> 


o 


CM 


o 


O 


o 


rH 


rH 


O 


rH 


O 


CM 


rH 


o 


rH 


o 


o 


rH 


O 


O 


O 


o 


O 


o 


rH 


o 


o 


H 


CM 


O 


o 


o 




0) 




O 


o 


LO 


o 


O 


O 


O 


O 


LO 


o 


O 


LO 


o 


o 


o 


o 


O 


LO 


O 


o 


o 


o 


LO 


o 


O 


o 


O 


O 


o 


LO 




=3 

rH 


P 




rH 


<r 




00 


O 


CM 


<r 


LO 


oo 


CM 


rH 


rH 


CO 


oo 


o 


oo 


CO 


rH 






CO 


ON 


CM 




oo 


CM 


CM 


rH 


oo 




cd 


MH 


a>> 


oo 


CO 


LO 


00 


LO 


ON 


VD 


oo 




00 


CO 


o 


CM 


CM 


xD 


CO 


rH 


ON 


in 


rH 


ON 


vX> 


a> 


CM 


oo 


LO 


O 


o 


CM 




> 


o 


CM 


LO 




vO 


<r 


LO 


vO 


vO 




rH 


CO 






vD 




LO 




vD 


xD 




vO 


vO 


CO 


LO 




LO 


CM 






LO 








CM 




CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 




cu 

MH ^H 

O 4=1 

O U 



(VlC0st^^N000>OHN(n^invON000>OHNcnstlAvON000>O 
H rl H rl -jrHiHrHiHrHCMCMCMCMCMCMCMCMCMCMCO 



244 



4J 



•H 0) 

> 

id a) 
o 

PM 



PM 



ooooooooooooooooooooooooo 



m m m 
o o o 



VAAAAAAAAAAAAAAAAAAAAAAAAAVAAA 

rHpDpMpMpMPMpMPMPMPMpMPMpMP*! pMpL(pL(pMPMpMPMPMPMp-,PMpMrHpL(pMpL( 

o o 
o o 



-a 

CD 

I 

C 

o 
u 



o 
o 

CO w 



CO 

< 

H 

S5 

CO 
CO W 

H 

CO 

W 
H 



CO 
55 

o 

H 
H 

<J 
SI 
H 



h a 



O 



55 
o 





55 




0) 




m 


vO 


o 




rH 




CO 


rH 




o 


w 


O 










o 


<f 


m 


<f 00 


<f 


O 


rH 




Pi 


CO 




I— 1 


M-l 
























H 




td 


Q 


CNl 


Q 


j—\ 


Q 


rH 


rH 


Q 


,__) 


Q 


rH 








> 




1 


1 


i 






r 


I 


1 






fx, 






























H 






























P 




Pi 




























O 








H 


















m 




u 


rH 






O 


















o 






























• 


i 


CO 




•rl 


























Ph 


55 


rH 


tH 


V 


m 


m 


m 


m 


m 


m 


m 


in 


V 






O 


*H 


0) 




o 


o 


o 


o 


o 


o 


o 


o 






8 


H 




> 


PM 




















o 


Pi 


H 


id 


0) 
























o 


U 






V 


A 


A 


A 


A 


A 


A 


A 


A 


V 


CO 




w 


o 
























H 


^* 


CO 


M 




rH 


PM 


PM 


PM 


PM 


PM 


PM 


PM 


PM 


rH 


CO 


o 




PM 




O 


















o 


w 


M 








O 




















H 


^ 


o 




























W 


H 


























4J 


H 


H 




























H 


<! 




























Pi 




0) 


























U 




4J 






























3 Q) 




vO 




O 


CO 


CO 


rH 


O 


O 


00 


co 




w 




H 3 










CM 


m 




co 


rH 


<t 


00 


CO 


rJ 




O H 




m 


00 




m 


<t 


r^. 


rH 


o 


o 


co 


w 


p 




en id 


m 
























p 


w 


rQ > 


o 


CM 


o 


rH 


o 


rH 


rH 


O 


rH 


o 


CM 




H 


PCS 






















































> 




55 


























CO 


o 






m 


o 


m 


in 


O 


O 


m 


O 


o 


m 




> 






S3 






















p 




w 


rH 




vo 


CM 




<t 


m 


O 




<r 


CT> 


m 


w 




►J 


td 


M-l 






00 


vo 




m 


rH 


vO 


CM 


00 


H 




> 


O 


i— i 


m 


co 


vO 


<f 


m 


r-. 


vO 


r^. 


CM 


< 










CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 



O O rH 



O O rH O 



O O O O O 
I I 



CO o o o 
I I I 



oooooooooooooooo 



m m m 
o o o 



AAAAAAAAAAAAAAAAVAAA 



<f CM 



m rH m vo o> cm 
o> m a> 
cm vo m 



o 
o 



rNCOCOCMOOCMCOH^CMrs 

t— imr— ic7>r^co^-r^cMT— io>vom 

r^rHinvOCOVO^fOCMr^CMriCM 



OOrHOOrHOOOOOOOrHrHrHCMOOO 



omoomoomoinininoooinoinmo 



0<frHCOCOvOVOrH 



o 



co 

vo cm m 



•st" vO vO CM 
CM I s ". 00 



rH CO rH 



H 0> H 



r^vovovomm^fmcMvor^ 



vO 



■S 
u 
td 
> 

m 
o 

QJ 

1 

55 




ooooooooooooooooooooooooooooo 



^ ^ CO 



O rO 

cd 

• *H 

» id 
> 



HCMcn<t^ONCO^OHCMCO<finvONCO^OHCM(0^invONCOO>0 

cococococococococo<t<f<f<f<r^<f<f<f<fininininininininini^ 



245 



9 

ERIC 



•H 



rd a) 
o 
p-i 



o 



in m m v m in in 

o o o o o o 

• • • p_i • • • 

A A A V A A A 

P-i P-i P-i tH P-i P-i P-i 
O 







0) 


4J 


o 




o 


o 


O 


oo 




o 








rH 


00 


CM 


<r 


O 






CO 






m 
















H 




cd 


o 


rH 


O 


o 


CM 


H 


tH 


o 


% 




> 




1 


























































O 


ON 












tH 








u 


rH 












O 












4-> 










• 












•H 




















& 


tH 


H 


m 


m 




V 


m 


m 


m 




o 


•H 


0) 


o 


o 


o 




o 


o 


o 


o 


H 




> 
















PC! 


H 


cd 


0) 
















O 


CJ 




►J 


A 


A 


A 


V 


A 


A 


A 




W 


o 


















^ 














tH 








O 














O 






H 














O 









0) 

•u 

3 0) 

H 3 N 

O tH 

co cd m 

rQ >. O 



rH CO CO tH O <t CO 

cri in <t o tH co 

00 H CM O O 00 CO 

O H O CO H tH O 



0) 
tH 

cd m 

> o 



in m m o o o o 

tH CT» CM CT\ h m CN 

tH CO ^ 00 O CM 

in <j- vo h n m m 

CM CM CM CM CM CM CM 

































0) 
















CO 














4-» 
















s 


















CO 










CO 


CO 


H 


g 




10 


to 


to 




4J 






(1) 


u 








XI 


a 


0) 


tH 








Oi 


*) 


•H 


H 


rQ 


bO 


GO 


CaP 


.-. 


H 






cd 


•H 


•H 


•rl 


CD 


a) 


CO 


X 


H 


CD 


0) 


0) 


£5 










PC 




!s 






VI 




rd 








rH 


CM 






> 


co 


CM 


H 


! 


1 


Itf 


H 


m 


<w 


m 


m 


tH 


m 


tw 


m 


o 


o 


o 


o 


O 


o 


o 


o 


0) 


u 


n 


n 


H 


n 


u 






0) 






0) 


0) 
























•a 


1 


■§ 


■i 


0 


1 


■a 






















jz; 




Jz; 


| 


53 


i 




rH 


H 


H 


rH 


H 


rH 


rH 




rd 


rd 


rd 


rd 


rd 


rd 


rd 




4-) 


U 


4J 


4-) 


4J 


4-» 


4-) 




o 


O 


o 


o 


o 


O 


O 




H 


H 


EH 


H 




H 


H 



0) 

m h 

O rQ 
rd 
• vH 
O H 

S cd 
> 



tH cm co <r m vo 

vO vo vO vc vO vO 



246 



u 



co 

W H 

<J 



o 
o 

co 
P-i 

o 
Pi 
o 

o 



o 

H 

co 

W H 

H g 

H 
CO W 

W PL, 

> • 

CO 
N > 



us w 
o 

CO 
CO 

< 



2! 

o 

H 
H 
<J 
N3 



*3 



CO 
CO W 



CO 

w 

H 



<3 w 
H O 

O 



S3 

o 























•H 








H 


H 






•H 


0) 








> 






ed 


CD 














O 
















P-l 








CD 
















H 


m 








o 






> 












Pi 








o> 








H 












4-» 








•H 








H 


H 


o 




•H 


CD 


H 






> 


H 




cd 


CD 


O 








w 




O 




CO 












P-l 












o 








H 








H 








<J 










0) 






1 


4J 










CD 




a 


rH 




N 




O 


H 






CO 


cd 


m 


w 


rO 


> 


o 


TH 


<! 














o 




CD 










P 


w 




H 








cd 


m 






> 


o 



J 

> 
o 



S3 



CD 

m H 

O 

td 

53 td 
> 



o 

ininininininininininininininininininininininininininv in m m 
oooooooooooooooooooopooooo ooo 

A A A AAAAAAAAAAAAAAAAAAAAAAAVAAA 
PhPhPhPhPhPhPhPhPhPhPhPhPhPhPhPhPhPhPh 

O 



cNOo^cNvocN^H^cN^cnHOOvoc^ln^c^^vocncoOoo^vOrlln 
vOHH^HCN(no.c^Hvoooocor<)^^c^corl^^^^^^^lcoH^l^^O 

HOOOHHHHOHHHHOOOHOHOHOO.^OHCNHHH 
III I I I I I I I I I I III I I I I I 



m 
o 



o 



m m m 
ooo 



inininininininininininmininininininininv mmmmmv 
oooooooooooooooooooo ooooo 

PM P-l * * * 

AAAAAAAAAAAAAAAAAAAAVAA AAAVAAA 

PhPhP-iP-iP-iP-iP-iO^PhP-iP-iP-iP-iP-iP-iP-iP-iP-iP-iP-iHP-iP-iP^ 

O O 



c^cnvD(N^oHo^o^l<roo<l'OlnOcNln^ooln^l^^vooc3^vo<|■^^c^ 
r^<f<finHv^cnHr^r^rHHrH^inc^ovOrHco^ocoocoo<-inooo 
(ninininr^vocco^c^vDr^on^oin^vDCNCNOr^^CNir^ocNHin 



OOOOOOHOOHO 



OOOHOHOCMHOHOHCNHHO 



ooinminmoooo. ininooinoinooooininoinooinom 

ino^OvovoH^coHcno^.^co^cNinvocovo^^csivocNN 

^^OvoHinHvo<fcor^Ovoocv»rsir-;<fHOHOin<fooHinH^"0 

^vovovovovovovovoincnvovor^vor^invo^r^^inin<fvoincoinin^ 

CsJCNCsJCsJCNlCslCsJCsJCSCNlCNlCsJCsJCslCslCNr^C^ 




cr»OHCNco<rmvor^oo<T»o 

__l rO rvi rvi rvi ^vi ^vi rs.1 f»o *s.i r<r\ 



■8 

■H 

4J 

o 
o 



247 



•H 

H tH 

•H <1) 

> 

id a 
o 

M 



o 



i— i 
o 



oooooooooo 



LP) •J") LP) LP) LP) LP) LP) LP) LP) LP) LP) LP) LP) LP) LP) 

ooooooooooooooo 



LP) LP) LP) 

o o o 



AAAAAAAAAAVAAAAAAAAAAAAAAAVAAA 

i 

O O 

O 



T3 

0) 

•H 

4J 

O 

o 



u 



2 
c 



"eg 



S3 



O 



ftf CO 
W H 

M 
P 



CO 
W . 



2 
O 
H 
H 
U 
W 
CO 

_ o 

W M 
H H 
H 

u 



CO W W 

H & PS 

> - 



co Jz;- 
> o 



p w w 
^ ° & 

H H <J 
O £ CO 

o 

CO W S 

CO ffi o 

.<! H H 
H 

td < 

HON 
H H 

CO <j 
lO t=3 P5 
H i-4 W 
CO PQ JZJ 
pq <] W 
HMO 



O 



<D -U 
H M-f 
> 



4J 

•H 

tH i-H 

•H CD 

^ > 

cd cd 

O 
PM 



0) 
4J 

H 3 N 
O H 

•5 > ° 



0) 
tH 

cd m 
> o 



•H 

cd 
> 

m 
o 

0) 

0 
cd 



ooOvoHHin^Hcnaicnc^ooOc^aicn^NH^DinNooino^-riOcsi 
ln•^•^vOlnrlHOfncoHoo^OlnooolnooHlO<■<■<■r^c^OMnln^ 

HOOOrHiHHTHHHcvJHOOOOHHOOHOHiHOrHCNiHTHtH 
I I I I J I I I I (111 I III I I I I I 



LP) 

o 



LP) 

o 



LP) LP) LP) LP) LP) LP) LP) LP) LP) LP) LP) LP) LP) LP) LP) IP) LP) LP) LP) LP) V LP) LP) LP} LP) LP) V LP) LP) LP 

oooooooooooooooooooo ooooo- ooo 

AAAAAAAAAAAAAAAAAAAAVAAAAAVAAA 

O O 



^inHOninoOoovocnNfoincnLOHooo^o^rivDH^Hoonin 
o^ooonc^loo^Hoo^^^4 0^D^«^\oc^lc^l(n^ln'v^OHc^lvoo^rn^ 
fnHfnLOoo^(^o^^nooo^Ocn^NLn^Hc^lcNo^cN^H^LOc^lcNln 

HOOOOOOrHOHHOTHOOOHOtHOcsiOHrHOr^C^HrHH 



LoooLno^jLoo-^oooLoooooLnoopooooLnLnOLno 
ooc^oocncocn0^^oo^oc^l^^^nvCLn^^l^voc^l^cN^Ho>^^Lr|oo 

(nH'<f^0^inOO^OOHCNvO^DOrnO>O^HOrlHCN|^HOiriOcnCv| 

^N^^in^vo^LnLnnLn^Di^^D^DLniriLn r^<rLPi-<r-<rr^LPicNLOLPi en 

CNCNCNIcNCNCNCnJCNCNCnJCNCNJcMCNCNCNC^ 




H 



M-l M-J M-l 
& & & £ 













h4 
H 








o 
















3 








H 
























O 




W 




w 




H 




Ph 








H 




W 




H 




O 




Q 




H 
























S3 




B 


CO 












H 


< 


TY 






1 


i 


TY 


H 




> 




o 


H 




O 


H 


o 


H 


k4 


H 


JZS 




w 


CO 


H 


PQ 


H 


S3 




CO 




H 




tH 




w 






o 


H 




Pm 


H 




o 


H 


H 


o 






p* 


S3 










o 


H 


H 


PM 




m 


m 


m 


M-l 


M-l 


m 


o 


o 


o 


O 


O 


o 




<H 






M-l 


M-l 
















o 

ERIC 



0) 

LW tH 

o ,o 
cd 

• >H 

O 

S cd 
> 



Hc^ln<rlnvD^coo^OHc^ln^■^^^coo^OHc^ln<rln^)^coo^o 



248 



4-) 
•H 

H H 

rO > 

Cd 0) 

^ iJ 
O 

u 

PM 



LO 

o 



o 



V LT) LO V LO LT> LO 

o o o o o 

pLt ► • P-« • • ♦ 

V A A V A A A 

H PM PM H PM PM PL, 

O O 



CD -P 

H m 

cd o 
> 



O 00 VD N CN O 

CN 00 H H r- i H 

» » • • • • » 

CM O O CN M H O 

II II 



4-) 
•rl 

H H 

•H CD 

r& > 

Cd (1) 

o 

PM 



o 



LO 

o 



v lo m v Lo lo lo 

o o o o o 

pL| • • PL* • • ■ 

V A A V A A A 

H H CH 

O O 



CD 

H 3 N 

O tH 

co cd m-j 

•9 > ° 
< 



r*^ cn o r>* o o 

CM co in ^ Q VO rl 
MM HHONVO 

CN O H CM o o o 



NJ > O 



cn 

CO W 



> O, 



<d 

•H 
LW 

o 



to o m m o lo in 
oo ro ro oo r> o> 

N ^ H CO ro js 

vo <f m lo lo 

i CN CM CN <N CN CN 



CD 
(0 
t3 



CO CO H 

CO CO (0 4-> 4J 

4J 4-) 4-> ^ rC! C 

^,£3^ W) 60 »H 

00 CO 00 »H »rl 

rl rl rl G) Q! W 

0) CD Q) 3 »d 

ts £ M 

HMO 
CO CN H I I 



^ IH 1W ^ »W »W «W 

o o o o o o o 



u 

CD 



II 



H M M M U 

CD CD 0) CD CD 

^ 

llils 



H H H H H H H . 

cd cd cd cd cd cd id 

4-) 4-> 4-> 4J 4_) 4-> 4-> 

O O O O O O O 

H H H H H H H 



ERLC 



CO 

<W H 

O rO 

cd 

o u 
& cd 
> 



H CN CO stf lo vo 
vO vO vO vO vO vO *0 



249 



•H 0) 

x> > 
o 

PM 



o 



«— I rH «— I 

o o o 
o o o 



o 
o 



LP) LP) O O O O 

o o o o o o 



m o o o o o o 
o o o o o o o 



■ ■ -g 

H H «— I «— I rH i— I i— I t— I C 

oomoooooo *h 

OOOOOOOOO -U 

c 

o 
o 



VVVVVVAAVVVVVAVVVVVVVVVAVVVVVV 

O O O O 

O O O 



W 

u 

w s 

P$ o 

W CO 

^ b 

Pn 

M <j 

P PM 

3° 

S CO 
PM 

C*< P 

o o 



o 



H 
co 
W 
H M 

H 



2 PC! Pm 

< O M 
fH 

CO CO 

W C£l p 

PM 1-3 
PM 
P 



P 
> 



O 



w 

PC 
H 

N CO 
> 

P 

W PC! 
H W W 



o 
o 

CO PM CO 

co m 
<S H S3 

o 

ffi Ph H 
Eh O H 
H fn<C 
£5 N) 
CO f-M 
CO W 



H 
CO 
W 
H 

P 



3 

> 



W 



h4 Pn 



25 S3 
55 O 





0) 


+- 




m 


m 






00 


O 


CO 


00 


O 


CM 


m 


CO 


m 


rH 




VD 




st- 




CO 


st 




o 


rH 


m 




CM 


st 


00 








r^. 


r^. 


rH 


VD 


CM 


CO 


CO 


r^. 


m 


m 


Sf 


o 


CO 


sr 


CO 




vO 




CO 


st 


rH 


kD 


st 


o 


CTN 




CM 


CM 


m 


r^. 




rH 


MM 


































































O 


st 


st 


00 


CM 


st 


CM 


rH 


rH 




sr 


VD 


m 


CM 


rH 


CO 


VD 


CO 


m 




m 


CO 


v£> 


o> 


rH 


CO 


st 


VD 


st 


m 


rH 


CO 


> 




1 


l 


f 


1 


i 




1 


1 


I 


I 


1 


I 


1 


I 


1 


1 


1 


I 


1 


1 


I 


I 


I 


1 


I 


I 


1 


1 


\ 


rH 
[ 


a> 


































































1— 1 












m 




m 














rH 
















rH 




















' — ✓ 


4J 










o 




o 


o 












O 
















O 






















•H 




rH 


rH 


rH 




rH 








rH 


rH 


rH 


rH 






rH 


rH 


rH 


rH 


rH 


rH 




rH 


rH 




rH 


rH 


rH 


rH 


rH 


rH 


o 


rH 


rH 


O 


O 


O 


V 


O 


V 


V 




O 


O 


O 


O 


V 


m 


O 


O 


O 


O 


O 


O 


V 


O 


O 


m 


O 


O 


O 


O 


O 


O 


H 


•H 




O 


O 


O 




O 






o 


O 


O 


O 


O 




o 


O 


O 


O 


O 


O 


O 




O 


o 


o 


O 


O 


O 


O 


O 


O 


H 


^ 


> 








PM 




PM 


PM 
















































O 




0) 






























































W 




rJ 


V 


V 


V 


V 


V 


V 


V 


A 


V 


V 


V 


V 


V 


A 


V 


V 


V 


V 


V 


V 


V 


V 


V 


A 


V 


V 


V 


V 


V 


V 


CO 


O 






































































PM 


Pm 


PM 


rH 


PM 


rH 


rH 


PM 


PM 


PM 


PM 


PM 


rH 


PM 


PM 


PM 


PM 


PM 


PM 


PM 


rH 


PM 


PM 


PM 


PM 


PM 


PM 


PM 


Pm' 


25 


PM 










O 




O 


O 












O 
















O 




















O 




























O 
















O 





















0) 
3 



rl 3 N 



0) 

O rH 

CO Cd M-I 
^» > O 



0) 

3 p 

rH 

MH 
> O 



•rl 

> 

m 
o 

§ 

23 



^^ninincM^<roofnofn^^cMvDO\HvD^H^Ho^HvDOH^ 

C0O0>0>r^HsfHOr^invDHsfHHCJ0C0SfHCJ0ini^OC0HHI^SfvD 

inincorsi^rsicsirivoiriNincsiriinN^inNincNi\DO\rioo^D <r m o> 



inominininoooinOininoinininoooinoinomoinomin 

CMOHNNNHaiH^inmsrO^vD^a\NOa\C0CMH^OHNC30H 
COiniOO^^NCO^C^HCSlVDOC^CsIin'sfinNOO^rlinON'sfVOOCVlvD 
C0VDC0nC0^COinCOin^in^^^Hr|VDO\VOCV|CS|^NVD00CnHNH 
H H CM H CM CM N H H H CM CM H H CM H H CM H CM H H CM rl 




14_|14_|14_|14_|14_|14_|14_|14-|14-| 
OOOOOOOOO 

g_ig_|M_|l4-|l4_|l4-|l4-|l4-|l4-| 



q-iq-iq-lM-iq-iq-lM-iq-iq-lM-lM-iq-lM-lM-l 



O rO 

* -i 
O U 

& cd 
> 



HNCO^invONW^OHCMCO^^vONCO^OHNcn , sfinvONOO^O 
rH rH t— I i— I i— I t— I i— I i— I i— It— ICMCMCMCMCMCMCMCMCMCMCO 



ERIC 



250 



•rl 

H t-i 

•H 0) 

.Q > 

o 

CM 



O O O 
O O O 



o 



o v m m o o o o 
o o o o o o o 



H 

o 



v m o o vooo v oomoooooo 
ooo ooo ooooooooo 



VVVVVVAAVVVVVAVVVVVVVVVAVVVVVV 

O O 1 O O O 

o OOO 



-a 

CD 

a 

•H 
4J 
C 

o 
o 



pa 
o 
ss 

w & 

W M 

H < 
P Pi 



13 
O 



W 

o 



W 
PM 
P 

o _ 
p$ w 
wow 

H 

¥i * 53 
woo 

H H M 

p$ H 

•M W <J 

H O 

P H H 

5 p4 

< O H 
H 

CO pn CO 

w w p 
D n 

> 



w 



N 

S3 



CO 

> a 
o 

w w 

O Ph 



o 
o 

WWW 

o 

ta ps h 

H O H 

w 
w w 

W ~" 

w 



N) 



w 



S3 

w 
o 

w 



vo H 

W 

i-3 O 



S3 
O 



0) 

3 0) 

O tH 

co rd M-r 

rQ > O 

<3 



CD 

P P 
tH 

> o 



0) 


+J 


CO 




vO 




r— ! 


VO 


CM 


H 




to 


tH 


CO 


CO 


tH 


00 


00 


tH 




m 


tH 


m 


o 


CO 


o 


o 


00 


tH 


CM 


o 


tH 






00 


vO 




on 






to 


-d" 




00 


O 


CM 


ON 


vO 


ON 


tH 


tH 


m 


m 


CM 


CO 


m 


o 


o 


CO 




rH 


on 


si- 


CM 


tH 


































































O 








CM 




CM 




tH 


vO 








CM 


tH 


CO 


vO 


CO 


m 




m 


CM 


vO 


o 


tH 


on 




vO 


CO 


m 


CM 


> 




i 


I 


! 


l 


i 


1 


I 


I 


1 


l 


1 


1 


1 


I 


I 


I 


I 


1 


I 


I 


I 


1 


tH 
! 


1 


l 


1 


1 


I 


l 


tH 
1 












m 




m 


m 












tH 
















tH 






























o 




o 


o 












O 
















o 
































































• 




















•H 




tH 


!— 1 


!— 1 




H 








H 


tH 


tH 


tH 






tH 


tH 


tH 


tH 


tH 


tH 




tH 


tH 




tH 


tH 


tH 


tH 


tH 


tH 


T— i 


r— ! 


O 


o 


o 


V 


O 


V 


V 


m 


O 


O 


O 


O 


V 


m 


O 


O 


O 


O 


O 


o 


V 


o 


O 


m 


O 


O 


O 


O 


O 


O 


•H 


0) 


O 


o 


o 




O 






o 


O 


O 


O 


O 




O 


O 


O 


O 


O 


O 


o 




o 


O 


o 


O 


O 


O 


O 


o 


O 












PM 




PM 


PM 




















































































































V 


V 


V 


V 


V 


V 


V 


A 


V 


V 


V 


V 


V 


A 


V 


V 


V 


V 


V 


V 


V 


V 


V 


A 


V 


V 


V 


V 


V 


V 


o 




PM 


PM 


PM 


H 


PM 


tH 


tH 


PM 


PM 


PM 


PM 


PM 


tH 


PM 


PM 


PM 


PM 








tH 






PM 


PM 


Pm 


Pm 


Pm 


PM 


PM 












O 




O 


O 












O 














o 












































O 
















o 























00 




o 




tH 










rH 


00 


00 


00 


rH 


m 


vO 






CO 




CM 




o 




00 


ON 


VO 


00 


m 


m 




CO 






CM 


o 


o 


tH 


a> 


ON 




CO 








ON 


o 


<t 


ON 




CM 




o 


CM 


CO 


rH 




m 


o 


o 




o 


m 


00 




m 




00 




tH 


o 


00 


<r 




CM 


m 


00 


vO 


CM 


ON 


ON 


vO 


o 


vO 


00 


tH 


tH 


CO 


o 


m 


m 


00 


CM 




CM 


CM 


tH 


vo 


m 




m 


CM 


tH 


m 




CO 


m 




m 


CM 


VO 


ON 


rH 


00 




vO 


<r 


m 


o 

tH 


o 


o 


m 


m 


o 


m 


m 


o 


o 


m 


m 


o 


m 


o 


o 


o 


o 


m 


m 


o 


m 


O 


o 


o 


m 


o 


o 


o 


o 


m 


CO 




o 


CO 


vD 




m 


<f 


vX) 




CM 


00 


tH 


CO 


tH 




CO 


rH 


00 


o 




O 


m 


tH 


00 


tH 


vO 


o 


tH 


tH 








00 


<f 




CO 


m 


CM 


o 


CO 


ON 


o 


00 


O 


m 


m 


VO 


vO 


o 


m 


o 




o 


ON 


CO 


m 


tH 


CM 


m 


CO 




00 


CO 


00 




CO 


m 


CO 


m 


ON 


m 




m 


<!- 


rH 


rH 


vO 


ON 




CM 


CM 


CO 




vO 


00 


CO 


tH 




tH 


H 


H 




CM 


H 


CM 


CM 


CM 


rH 


tH 




tH 


CM 


CM 


tH 


tH 


CM 


rH 




tH 


CM 


rH 




CM 




tH 


rH 


CM 


rH 






w 

H 
P 

- ?3 
Q W PM Ph 

M-lM-|iwM-lM-lM-|iWM-lM-l4-(M-lM-lM-lM-lM-l 

ooooooooooooooo 



o 

H 

S3 

> 
M 
H 

O 

W H 
O H 
W H fc 
> > W 
M & H 
WO 



ON 

m 

ft 

O 



W CO 



O 
H 

H 
W 
w 
w 

<J 

o 



CO 
CD 
tH 

rQ 
CTJ 

H 
U 

> 
o 





CD 




tH 


o 


ab 




•H 




M 


1 


Va 



rHcMco^invor^cooNO'tHcMco^invor^oooNOrHcMco^invor^ 



i 



251 



•H 



id o 

O 

M 
ft 



rH H H H H 

O O O lo lo o o 

o o o o o o o 



V V V A A V V 

P_( ft ft ft ft pL^ ft 



0) 






O 


o 


LO 


LO 




00 






rH 


00 


00 








o 


H 


m 














• 


id 


o 


O 


d 


00 


d 


d 


d 




> 




H 
1 


H 

1 


I 


i 


1 


rH 
1 


rH 

r 



4J 
■H 

•H 



id cd 
O 

u 

ft 



H.H H rH r-r 

o O O lo to o o 

o o o o o o o 



V V V A A V V 
(1h ft Ph ft 



CD 
4J 

H 3 N 
O H 

co lw 



o» omtio cn ctn 
^ fn N en o o> H 

o> o> o o o> o 

rH 



CD 
rH 

id m 
> o 



LO lo O to O O lo 
00 N O CN CO CM 

^« o r\i o ro rn nI - 
n cn <f n cn h 

CM CM 



CD 
co 















4J 


£3 














B 


















CO 










CO 


CO 




e 




CO 


CO 


CO 


-u 


4J 






CD 


.p 


4J 


4J 


,c 


•C 




CD 


rH 


42 






ioQ 




H 




oo 


00 


60 


•H 


•H 






ia 


•H 




•H 


CD 


CD 


CO 


H 


CD 


CD 


CD 


& 


& 






M 


ras 


& 


& 






M 




cd 








H 


CM 






> 


CO 


CM 


H 




1 




H 




LH 


m 


LW 


4-1 


M-l 




m 


0 


o 


0 


0 


O 


0 


0 


0 


(D 








M 




u 








CD 




0) 


CD 


CD 


CD 


















S3 


| 


















2; 


53 


53 


52; 


53 


53 




rH 


H 


H 


H 


rH 


rH 


H 




cd 


cd 


id 


cd 


crj 


cd 


cd 




4-) 


4J 


4J 


4J 


4-» 


4J 


4J 




0 


0 


0 


0 


O 


0 


0 




H 


H 


H 


H 


H 


H 


H 



ERJC 





CD 


LH 


rH 


0 


ab 


i 


•H 


o 


U 


53 


Va 



H CM CO ^ VO N 
O VD AO vO VD VO 



252 



•H 0) 
Xi > 

O 



o 
o 



o 



o 
o 



P4 

V V 



i— I PM i— I 

o o 
o 



to 
o 



vomintnomom v m v 



oooooooo 

VAAAVAVA 
PMPMPMPMPMPMPMPM 



V A 
i— I PM 



m 
o 



o v 
o 

• PM 

V V 



PM 

V 

i— I PM i— I 

O O 

o 



m i— i 
o o 



o o v o o m 
o o ~ 



o v v 



li v II 

PM PM PM 



O 
O 



o o o o 

• • • • PM PM 



V V 
PM 



A V V V 

pL, pLt H H 

O O 

• O 



O O O 
O O O 



II V V 

Pu Pu Pu 



0) 
4J 

o 
o 



w 

o 



H 
P 



o 
co 

H 



O i— I 



CO 5 
PM O 
P H 

S H 

o w 
co 

§s 

H O 
f* H 
W H 
H <J 
H O 
ftf H 
O Pk 
H 
W H 
CO 

• H 

N CO 
> & 
O 



o 

CO 
H 
CO 

w 

H 
4J 



ro 
W 
P 



H 

O 

o 

CO 
CO 

< 



W CO 

o 

ffi Ph H 

H O H 

H <J 

CO H 

CO W q 

H h3 <3 



5§ H 



H 



o 



CD 
4-> 

3 0) 

H 3 N 

O i— I 

CD td <W 

*q > o 



0) 



td m 
> o 



cd 
•H 

id 
> 

m 
o 



0< S 

O U 
> 



0) 


4J 


co 


00 


CO 


CM 


O 


00 


o 


00 




o 


iH 


o 


CM 


ON 


CM 


vd 


vD 


00 


r— 1 


1— 1 


m 


CO 


vD 


vD 


CM 


CM 




as 




CM 






o> 


i— 1 


00 


o 


i— 1 


<f 




CO 


vd 


m 


CO 


vD 


m 


CM 


O 


1— 1 


o 


CO 


CM 




00 


r-* 


o 


o 


vD 


<!■ 


i— 1 


CM 


vO 


CO 


1— 1 
































































id 


o 


CO 


CO 


<f 


CM 


<f 


1— 1 


1— 1 


o 


<r 


i— i 


m 


i— 1 


CM 


i— 1 


CO 


Si" 


CM 


CO 


vD 


CO 


CM 


CO 


00 


00 


vD 


CM 


CO 
I 


CO 


CO 


C\ 


> 




I 


I 


1 


1 


I 


I 


1 


1 


I 


l 


i 


1 


r 


I 




t 


I 


1 


f 


I 


1 


1 


1 


1 


I 


I 


I 


I 


1 








1— 1 










m 












i— 1 








m 








i— 1 










m' 




i— l 












o 










o 












o 








o 








O 










o 




.o 






































































•H 




1— 1 




1— 1 




i— 1 








i— i 




i— i 








i— 1 


1— 1 




1— 1 


i— 1 


1— 1 




1— 1 


1— 1 




i— 1 




i— 1 




i— 1 


1— 1 


i— 1 


1— 1 


o 


V 


o 


m 


o 


m 


V 


m 


o 


m 


o 


m 


V 


m 


o 


o 


V 


o 


O 


o 


V 


o 


o 


m 


O 


V 


o 


V 


o 


o 


•H 


0) 


o 




o 


o 


o 


o 




o 


o 


o 


o 


o 




o 


o 


o 




o 


O 


o 




o 


o 


o 


o 




o 




o 


o 








PM 


























































td 


5 


































































V 


V 


V 


A 


V 


A 


V 


A 


V 


A 


V 


A 


V 


A 


V 


V 


V 


V 


V 


II 


V 


V 


V 


A 


V 


V 


V 


V 


V - 


V 


o 




PM 


1— 1 


PL. 


PL. 


PL. 


PLj 


i— I 


PL. 


PL. 


PL. 


PM 


PM 


1— 1 




PL. 


PL. 


i— I 


PL. 


PL. 


PL. 


1— 1 


PL. 


PL. 


PL. 


PL. 


i— 1 


PL. 


1— 1 


PL. 


PL. 


Put 






o 










o 












o 








o 








o 










o 


o 












o 






















o 
















o 














o 












• 






















• 
















• 














• 







1— 1 


CO 


00 


\D 




CM 


\D 




i— 1 




O 


CM 


CO 


00 


CO 


<r 


CM 




O 


o 


i— 1 


O 


i— 1 


o 




00 


VD 


<r 


CM 


CO 


l\ 


o 


<f 


CO 


m 






i— i 


<f 


<r 


Sf 


CM 




\£> 


CO 


CO 


i— 1 


CO 


00 


as 




O 


00 


o 


00 


CO 




00 


as 


m 


00 


1— 1 


CM 


00 


H 


CO 


o> 


o 


00 




<r 


as 


VD 


i— 1 


<r 


CM 


<r 


m 


O 


CM 


vD 


m 


VD 


o 


as 


CO 




i— 1 


VD 




CO 


CO 


m 


i— 1 


<r 


H 


i— 1 


o 


<r 


1— 1 


m 


H 


CM 


H 


<f 


m 


CM 


CO 


VD 


CO 


CM 


<r 


00 


o 


VD 


CM 


CO 


CO 


CO 


00 


m 


m 


o 


m 


o 


O 


m 


o 


m 


o 


m 


o 


m 


O 


o 


o 


o 


o 


m 


o 


O 


o 


m 


o 


o 


O 


o 


o 


m 


o 






CM 


i— l 


CM 


\D 








CM 


i— i 


00 


o 


\D 


m 


o 


as 


vD 


o> 


00 


i— 1 


o 


CO 


00 


CM 


i— 1 


00 


1— 1 


<r 


00 


r^. 


<r 


m 


H 


CO 


CM 


00 


CO 


00 


00 






<f 


CM 




m 


o 


m 


i— l 


o 


CM 


\D 


m 


CO 


00 


o 




CO 


<r 




i— 1 


CM 




m 


o 


\D 


<r 




00 


CO 


sr 


CO 


<!• 


\£> 






<r 


CM 


<r 


CM 


CO 


00 


vD 




1— 1 




as 


CO 


i— i 




CM 


CM 


i— I 


CM 


CM 


CM 


CM 


CM 


1— 1 


CM 


i— i 


CM 


CM 


CM 


i— I 


i— 1 


CM 


CM 


i— i 


CM 


CM 


i— 1 




CM 


H 


CM 


i— 1 


CM 


CM 






I4_|l4_|l4_|l4-|l4-|l4-|l4_|l4_|l4-|l4-|l4-imi4_l 



cMrn^invDNoocnOriNn^^^Nooc^OHMrn^invoNooc^o 

HHHHHHrHHHHCMCMCMCMCMCMCMcMCMCNJcO 



253 



S3 
O 



P* CO 
W M 
Ph 
Ph 



3 



CO 



o 

W CO 



Ph 
o 

CO 
H 
CO 

w 

H 



CO 

w 

I 



o 

H 

P^ 

H 
H 

O 

w 

P 
P 



N CO 
> 

P 

M O P-j 

o < 

CO W CO 

CO X 

< H 



S3 P* 
H O 



S3 

o 



Ph 



3 

N 
CO H 
CO W 
H rJ 
CO PQ 
<1 
H 



H 



vD 

S3 
O 



W 
IS 
W 
O 

w 
u 

H 

Ph 
O 



cd a) 

o 

Ph 



Q) 4J 

iH MH 

cd O 
> 



O V 

o 



o o 



ovommmomomv mv 
o oooooooo o 



m 
o 



o m v 
o o 



O rH 

o o 



o o m o 
o o o o 



VVVVVAAAVAVAVAVVAVV II V V V A V 

o o o o o o 

o o o o 



o o 

V 

Ph 

V 



o o 



I I I I I I I I I i I i i I ! i i I i I i I I I I I I I ! i 





I— 1 


I— 1 


V 


o 


o 




o 


o 


Ph 






V 


V 


V 


I— 1 


Ph 


Ph 


o 






o 








vO 


m 


CO 


vO 




CM 


CO 





rH 










rH 
















m 






rH 








m 






rH 


rH 










m 




rH 










>•> 






O 
















o 






o 








o 






O 


O 










O 




O 










4J 




































































•H 




iH 




rH 




rH 








rH 




rH 








rH 


rH 




rH 


rH 






rH 


rH 




rH 




rH 




rH 


rH 


o 




rH 


(— 1 


O 


V 


o 


m 


O 


m 


m 


m 


O 


V 


O 


m 


V 


m 


O 


O 


V 


O 


o 


V 


V 


O 


O 


in 


o 


V 


O 


V 


o 


O 


H 




•H 


0) 


O 




o 


o 


O 


o 


o 


o 


o 




O 


o 




o 


O 


O 




O 


o 






O 


O 


o 


o 




O 




o 


O 


H 










Ph 




































Ph 


Ph 










Ph 




Ph 






o 




cd 
































































w 






>-H* 


V 


V 


V 


A 


V 


A 


A 


A 


V 


V 


V 


A 


V 


A 


V 


V 


V 


V 


V 


V 


V 


V 


V 


A 


V 


V 


V 


V 


V 


V 


CO 




o 








































































Ph 


rH 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


Ph 


rH 


PL, 


Ph 


rH 


Ph 


Ph 


Ph 


rH 


Ph 


Ph 


rH 


rH 


Ph 


Ph 


Ph 


Ph 


rH 


Ph 


rH 


Ph 


Ph 


S3 




Ph 






O 
















o 






o 








O 






O 


O 










O 




O 






o 










O 






















o 














o 


o 














o 






CAT I 










• 


























































H 


0) 


































































Ph 


4J 


































































H 




0) 




r^. 


sr 


o 


<T 


00 


CN 


O 


O 


vO 


m 




CM 




00 


CO 


CO 


CO 


vO 


rH 


CO 




<r 


m 


O 


rH 






o 


<r 


VO 


H 


rH 




N 


00 


00 


o 


00 


CO 




CO 


O 


CO 


CO 




m 


CO 






O 




rH 


vO 


<r 


rH 


00 


o 


O 


00 




m 


vD 


CM 




CO 


o 


H 




00 


vO 


K co 


00 


rH 


CO 


<J\ 


O 


00 


CO 


o 


00 


vO 


rH 




CM 


CO 


m 




rH 


vO 


m 


00 


O 




CO 




rH 




m 




CO 


cd 


m 






























































THE J 


Ab 


> 


o 


CO 


CN 


m 


rH 


<<r 


i— 1 


rH 


O 


<r 


CM 


LO 


rH 


CM 


rH 




m 


CM 


CO 


m 


CO 


CM 




00 


o 


vD 


CM 


CO 


CO 


CO 


CO 




0) 




o 


m 


m 


m 


o 


O 


O 


O 


O 


m 


o 


m 


O 


O 


m 


o 


O 


o 


m 


O 


o 


o 


o 


o 


O 


O 


m 


m 


m 


o 








E3 






























































o 




I— 1 






CO 


a> 


m 


sr 


<o 


rH 


00 


o> 


o 


o\ 


o 


o> 


m 


m 


rH 


<J- 


00 


ON 


rH 


<■ 


a> 


<r 


00 


CO 


m 


CM 


CO 




CO 






cd 


m 


r^. 


r— 1 


CO 


o 


co 


CM 


o\ 


CO 


00 


rH 


m 


CT> 


CO 


CM 


CO 


m 


rH 


m 


CO 


CO 


CM 


CO 


rH 


CO 


CM 


o\ 




CO 


CO 


CM 


w 




> 


0 


iH 


CO 


vo 


m 


o 


vO 






00 


CO 


m 


CO 


<r 


VO 






<d- 


CM 


<r 


CM 


CO 


00 


vO 


r^. 


CM 


CO 




CO 


rH 


m 










CN 


CM 


H 


CN 


CN 


CM 


CM 


CM 


rH 


CM 


rH 


CM 


CM 


CM 


rH 


rH 


CM 


CM 


rH 


CM 


CM 


rH 




CM 


rH 


CM 


rH 


CM 


CM 





^> 
cd 

•H 
Jh 

cd 

> 

m 
o 






o 




o 




of 


of 


of 


of 


of 


wf 


wf 


wf 


wf 


wf 




w s 

CO H 

W Ph 

Pi! F=h 

Ph < 

W H 

pd CO 



MHMHMHMHMHMHM-im 
OOOOOOOO 

^^^^^^^^ 



(1) 

a 

•H 

o 
o 



MH MH MH 
15 & > 



m m mh ih *H 4h 
^ 3 S 3 





a) 


m 


rH 


o 


ab 


• 


♦H 




>H 




Va 



H^ro^^^Nco^OHCNr^^invDNcoo>OHcNfo , <rLn<DNooo>c 



254 



>> 

•H 

H H 
•rl 0) 

r£ > 

cd CD 

o 
u 



o o o in in o o 
o o o o o o o 



V V V A A V V 



Q) -U 

H m 

cd o 
> 



on cr» oo o o vo o\ 

CN O O O vO CO 

r^. oo o o oo H 

I I I I H 

I 



4J 
•rl 

H H 
•H d) 

£ > 

cd <D 
pJ 

O 

u 



o o o in in o o 
o o o o o o o 



V V V A A V V 
p-i p-i p_i 



CD 
4J 

d CD 

rH d N 

O H 

CO Cd M-l 
> O 



00 00 <f o O VO VO 

00 VO H O O O 

O O O fO 00 

o o oo oo 



CD 

cd m-i 
> o 



o o in o o o o 

<f <f cm oo oo o\ vo 
vO *<t vO ro ro vO vO 
o\ ^ ^ in <f 

CM CM 



(D 

co 

ED 















cDv 


CO 










CO 


CO 


H 


g 




CO 


CO 


CO 


u 


4-J 






CD 


4J 




u 




& 




CD 


H 


& 




& 


00 


00 


•S 


H 




60 


00 


00 


•H 


•H 








•H 


•rl 


•rl 


CD 


CD 


CO 




•3 


CD 


CD 


CD 


& 






tt 








& 










cd 








H 


CN 


o 


•a 


> 


CO 


CN 


H 


I 


1 




M 


m-i 


m-i 


M-l 


M-l 


M-l 


M-l 


m-i 


M-l 


o 


o 


O 


O 


O 


O 


o 


O 


CD 






U 


U 


U 




u 




<D 


(D 


CD 


CD 


CD 


CD 


CD 














& 






I 


I 


■§ 


3 






















I 








1 












H 


H 


H 


H 


H 


pH 


pH 




cd 


cd 


cd 


cd 


cd 


cd 


« 




4J 


4J 


4J 


4J 


4J 


4J 


4J 




O 


O 


o 


O 


O 


O 


O 




H 


H 


H 


H 


H 


H 


H 



ERIC 



CD 

M-l H 

o rn 
cd 

o u 
& cd 
> 



H m n Sf lO vO |s 
vD vO vD vO vO vO vO 



255 



rH m 

•h a) 

td a) 
o 



m rH 

o o 



m 
o 



iH 

o 



h h m 
o o o 



o 



m 
o 



in v o m m m m m v 
o er o o o o o 
• p_i ••••»• p_( 

AV II A A A A A V 

P-irHPMPMPMPMPMPMrH 
O O 

o 



o v ommmv mv v vmominovomv o 
o oooo o ooooo oo o 

• p_i • • • • p_( • p_( p_i p_( • * • » • p_( • • Pm • 

II V IIAAAVAVVVAVAAVVVAVV 
FMHFMFMFMPkHFMHHH^ftPM^FMHC^FMHpM 



o o o 
o o 



T3 

CD 

•H 
O 

o 



Q) 4J- 

rH U-J 

td o 

> 



rH rH 



•H 

rO 



td Q) 
o 
Pm 



3 a) 

O rH 

CO td M— » 

> O 



0) 
rH 

td *4H 

> o 



ON 




rH 


O 


m 


CN 


on 


ON 


CO 




rH 


CO 


on 


O 


vO 


ON 


CN 


CO 


ON 


O 


m 


CN 




o 


VO 


VO 


ON 


CN 


CO 


O 


m 


m 


CO 


in 


rH 


VD 


rH 


rH 




m 


<r 


<T 


<r 


O 


m 


O 


CO 


rH 


vO 


O 


rH 


CO 


m 


o 


vO 


•si" 






CN 


CO 


rH 
1 


CN 

i 


co 


o 
1 


rH 
1 


O 
1 


O 
1 


rH 
1 


CN 
I 


co 
l 


CN 
1 


co 
1 


o 


o 


o 
1 


CO 
t 


rH 
I 


CO 
1 


CN 
1 


CN 
t 


o 
1 


CO 
1 


rH 
1 


rH 
1 


CO 

1 


CN 
1 


CO 
1 


rH 
1 


CN 
1 


<r 
I 




rH 














rH 




m 










r- { 




, — | 


in 


in 












, | 


r~i 




in 






o 














o 




o 










o 




o 


o 














Q 


Q 




Q 








rH 














rH 




rH 




















rH 






rH 










rH 


m 


V 


O 


m 


m 


m 




in 


V 


O 


V 


O 


m 


m 


m 


V . 


m 


V 


V 


V 


m 


O 


m 


m 


O 


V 


V 


m 


V 


O 


o 




O 


o 


o 


o 


o 


o 




o 




O 


o 


o 


o 




o 








o 


O 


o 


o 


O 






o 




O 




PM 














PM 




PM 










PM 




PM 


PM 


PM 












PM 


PM 




PM 




A 


V 


V 


A 


A 


A 


A 


A 


V 


V 


V 


V 


A 


A 


A 


V 


A 


V 


V 


V 


A 


V 


A 


A 


V 


V 


V 


A 


V 


V 


Pm 


H 


PM 


PM 


PM 


PM 


PM 


PM 


rH 


PM 


rH 


PM 


cn 


PM 


PM 


rH 


PM 


rH 


rH 


rH 


PM 


PM 


PM 


CM 


PM 


rH 


rH 


PM 


rH 


PM 




O 














o 




o 










O 




O 


o 


O 












O 


O 


O 




O 














o 














O 




O 
















O 


o 










00 


CN 


CN 




m 


CO 


rH 




CN 


CO 


m 


ON 




ON 


O 


CO 




o 


<r 


m 


m 


ON 


O 


rH 


O 


rH 


rH 


ON 


CN 


VD 


vD 


on 


rH 


co 




CO 


CO 


rH 


vD 


rH 


co 


vO 


CO 


co 


tn 


co 


VO 


ON 


vo 


CN 


ON 


m 


o 


rH 






m 


rH 


<!- 


<t 


VO 


VD 




co 


rH 


VO 


vO 


VO 


CO 


<r 


m 


rH 


CN 


o 


ON 


m 


o 


CO 


CO 


CO 






o 


<f 




CN 


CO 


rH 




rH 


CN 


co 


o 


o 


rH 


o 


rH 


CN 


CO 


CN 


CO 


O 


o 


rH 


CN 


rH 


CO 


CN 


CN 


o 


co 


rH 


rH 


co 


CN 


CO 


rH" 


CN 




O 


O 


O 


m 


m 


O 


m 


m 


O 


m 


O 


m 


m 


O 


in 


O 


m 


o 


m 


O 


o 


o 


m 


o 


o 


O 


o 


o 


m 


m 


o 


m 


CN 


tn 


vd 


CO 


CO 


vD 


vD 


co 


CO 


CO 


CN 




<r 


rH 


<f 


vO 


CO 


CO 


vD 


<f 


vO 


rH 




O 


r-. 


CN 


CN 


CN 


<t 


m 


rH 


CN 


m 


on 


CO 


m 


CN 


CO 


CN 


o 


rH 


o 




<t 




ON 


<f 


CN 




ON 


vO 


o 


VD 


CO 


<r 


ON. 


CO 


o 


<J- 


rH 


CO 


VO 


m 


m 


VD 


m 


rH 


co 


rH 


ON 








O 




o 


rH 


CN 


vO 


CO 


CO 




co 


rH 


ON 


<3- 


CN 


m 


CN 


CN 


rH 


CN 


CN 


CN 


CN 


CN 


CN 


rH 


CN 


rH 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


CN 


rH 


CN 


CN 


rH 


CN 


rH 


CN 


CN 


rH 



Pm N 



td 
•H 

JM 

> 

O 
0) 

I 

53 




tMl4_j"4-lMHl4-Jl4-jmi4-Jl4-J 

ooooooooo 



iWiWiHiHiWiHiHiHiWiHiHiH^^miWiHiH 
OOOOOOOOOOOOOOOOOO 



o o 



0 



ERJC 



a) 

M-? rH 

o ,o 
id 

O H 

jz; td 
> 



HcNrn<finvO|N.cooNO Hrsjro<fin^DNoo^oHcN co^rmvor^cooNO 

i — I H i — I i — I i — i rH i — ii — I i — I i — fCNCNCNCNCNCNCNCNCNCNrO 



256 



o 

w o 
pi co 

W H 

< 



00 



o ^ 
o 

CO o 

pti H 

P H 

o o. 

pi W 

O CO 



o o 

W H H 

H Ph H 

W <J 
4JHU 

H H 

p ^ 

2 O H 

< H 

Pi CO 

co w p 

W h 

p PM 

> • H 

CO 

> 



H P 
co W 

H P4 

CO 
CO W 
H >J 
co pq 

£3 



S3 
o 

w 



CO 

o 

H 



S3 
o 



o 



m i— i 
o o 



o o 



o o o 



m 
o 



m 
o 



m 
o 



•rl 




rH 


H 


•H 


0) 


X) 


> 


rr» 


0) 






^< 




H 




PH 




0) 








rH 


m 




o 


> 








4J 




•H 




H 


t— 1 


•H 


0) 




> 


id 








o 













0) 
4J 

3 CD 

rH 3 N 

O rH 

CTj Cd m 
> O 



CD 

3 P 
rH 

cd MH 
> O 



cd 
> 



m v v in in in in in v o 
o ■ o o o o o o 

• p-i p-t PM • 



v v m m m o m v v vino 
o o o o o o o 

P-Ip-I pL|pL|pL| • • 



m o v 
o o 



o m v o 
o o o 

. . PL, . 



AVVAAAAAVVVVAAA 11 AVVVAVVAVVVAVV 
P-« t — ! t — ! P-t P-i P-t P-i P-t t — ! P-c rHrHP^P^P^P^pMrHrHrHpMpMrHpMPMrHP-iP-ii— Ip-i 



O O 
• O 



O 
O 



O O 
O O 



o o o 
o o o 













CM 


os 


m 




rH 


CM 


CO 




as 








CO 




rH 


O 


o 


m 


O 








m 


00 


i > 




CO 


i— I 




OS 


OS 


rH 




OS 


00 


rH 


rH 


rH 


CM 




CM 


CO 


rH 


OS 


o> 


O 


OS 


CM 


O 




o 




r*-« 


O 


CM 


H 
1 


CM 
| 


CO 

1 


o 
i 


o 
1 


o 
1 


O 
1 


rH 
1 


CM 
I 


CO 
1 


CO 

1 


CO 

1 


o 


o 

1 


o 
1 


CO 

1 


t— I 
I 


CO 

1 


CM 
| 


CM 
[ 


o 


CO 

I 


CM 
| 


!— | 
| 


CO 

I 


CM 
I 


CO 

1 


H 
l 


CM 
i 


m 
l 




i— 1 














rH 




rH 










rH 




rH 


rH 


rH 






m 






rH 






m 






O 














O 




O 










O 




O 


O 


O 






o 






O 






o 








rH 














rH 




rH 




















rH 






rH 




rH 






rH 


m 


V 


O 


m 


m 


m 


m 


m 


V 


O 


V 


O 


m 


m 


m 


V 


m 


V 


V 


V 


m 


O 


V 


m 


O 


V 


O 


m 


V 


O 


o 




O 


o 


o 


o 


o 


o 




O 




O 


o 


o 


o 




o 








o 


O 




o 


O 




O 


o 




O 




PL< 














P4 




PM 










PM 




PM 


PM 


PH 


















Pn 




A 


V 


V 


A 


A 


A 


A 


A 


V 


V 


V 


V 


A 


A 


A 


V 


A 


V 


V 


V 


A 


V 


V 


A 


V 


V 


V 


A 


V 


V 




1— 1 


PL| 


PH 


PH 


PU| 


PL| 


PL| 


rH 


PL< 


rH 


PM 


PM 


PM 


PM 


rH 


PM 


rH 


rH 


rH 


PH 


PM 


rH 


PM 


PH 


rH 


PH 


PH 


rH 


PL< 




o 














o 




O 










O 




O 


O 


O 






o 






O 






O 






o 














O 




o 










o 




o 


o 


o 












O 












1— 1 


o 


CO 




a\ 


00 


m 


CM 


o 


o 


a> 


<f 


CM 


CM 




r>. 




vD 


o 


CO 


00 


CO 


o 










O 


CO 




00 


CO 




CO 




00 


o> 




00 




00 


CM 


rH 


m 


o 


<f 


OS 


CO 


CM 


o 






o 


H 


CO 




vo 




rH 




r^. 


m 




os 


H 










o> 


<r 


CM 


CO 


CM 


o 


<f 


o 


00 




CO 


CO 


<r 


o 






CM 


CO 


O 


H 


H 


CN 


CO 


o 


o 


rH 


O 


rH 


CM 


CO 


CM 


CO 


O 


o 


rH 


CO 


rH 


CO 


CM 


CM 


o 


CO 


CM 


H 


CO 


CM 


CO 


H 


CM 


m 


m 


o 


m 


m 


m 


m 


O 


O 


O 


m 


m 


m 


O 


o 


o 


o 


O 


o 


m 


m 


o 


o 


o 


O 


m 


O 


o 


o 


O 


o 


CO 


00 




o 


<t 


CM 


m 


m 




m 


vo 






rH 


m 




o 




OS 


H 


o 


OS 


o 


H 


00 




o 


00 




m 




CM 




CM 


co 


os 


CM 


m 


00 






rH 


O 


o 


CM 


o 




00 


rH 


<^ 


00 


rH 


o 


O 


o 


OS 


CO 


CO 


OS 


o 


si- 


!— 1 


00 


V0 


m 


m 


V0 


m 


O 


00 




ON 






<f 


o 




o 


O 


H 


AO 


a> 


H 




00 


H 


OS 




CM 




CM 


CM 


H 


CM 


CM 


CM 


CM 


CM 


CM 


rH 


rH 


rH 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


CM 


H 


CM 


CM 


H 


CM 


H 


CM 


CM 


H 




53 
O 
H 
H 
< 



0 JZJ 

CO H CO 
HfnOtJ 

pi h5 

pu, <? W H 

W H CO ^ 

p^ CO p CO 




ooooooooooooooooooooooooooooo 







CO Q 


CO 




rJ 




CO 


CO 


rJ 






CO 


r- 














iO H 








H 


g 














g 




CO 


a 


H 






H 


O 


p 


o 




P 


o 


PM 






CO 


CO 


a 


q 




w 


w 


pi 




Pi 




pm 




m 


MH MH 


o 


o 


C 


o 




4H 










% 





5 

H H 
CO 



PM 



> 
H 

Pi 



o 

H 

> 
H 
H 
O 

w 
a 



m 

-a 

o 

rH 
W CO 



o 
H 



H H 
p W 

PL| CO 



CO 



-a 

■rl 

a 
o 
o 



0) 

MH rH 
O 

cd 

O m 
S cd 
> 



Hcsco^invONcoc^OHc^cn^in^ONoooNOHNn^in^ONco^o 
n^nnnnnnn^^^<f<f<r^f^<r<fininininin 



257 



0 

ERIC 



•H CD 

rO > 

cd a) 
o 

PL, 



o o v m m o o 

o o o o o o 

• • P-| • • » • 

V V V A A V V 
fHpL|HpL|pL|pL|pL| 

o 
o 



CD 4J 

h m-i 

cd o 
> 



cm oo o in in r**. cm 
00 r*. -st <1- <1- <!■ 



-<j" -<j" cm o o -si" in 
I I I I I I I 



u 

H H 

> 

cd a) 

o 
u 



m 
o 



o o v m m o o 

o o o o o o 

• • p_i • • • • 

V V V A A V V 

(1| H (li 
O 



CD 

u 

H 

O 



CO 








00 


m 


o 


ON 


O 




N 


i— i 






CO 


o 


CO 




i— I 




00 




•st 


<t 


o 


m 




cd 


M-I 
















> 


O 


-d- 


<t 


CM 


O 


o 


-ci- 


<!■ 


0) 




O 


o 


m 


m 


o 


in 


o 




P 
















H 




H 




00 


CM 


00 


<t 


o 


cd 


m-i 






00 


o 


CO 


m 


<j- 


> 


o 


<!■ 


m 


o 






m 


<t 






H 


H 


CM 


CM 


CM 


H 


H 



H O 



W 









0) 




CO 










0) 


CO 











CO 


CO 


H 






CO 


CO 


CO 




4J 




Id 


(1) 


4J 


4J 


4J 






P3 




H 


X! 






00 


00 


•H 


h 




00 


00 


00 


•H 


•H 






cd 


•H 


•H 


•H 


CD 


CD 


CO 




•H 


CD 


(D 


CD 


& 


& 


XI 


CD 






& 


& 








"d 


cd 








rH 


CM 




C3 


> 


CO 


On) 


H 


1 


1 




H 


m-i 


M-I 


M-I 


m 


M-I 


M-I 


M-I 


M-I 


o 


O 


O 


o 


O 


O 


O 


O 


CD 


u 






U 


U 








CD 


CD 


CD 


CD 


CD 


CD 


CD 


1 


rO 


















1 


•§ 




"I 


•§ 


I 




















S3 


S3 














al 


al 


al 


al 


al 


al 


al 




4J 


4J 


4J 


4J 


4J 


4J 


4J 




O 


O 


O 


O 


O 


O 


O 




H 


H 


H 


H 


H 


H 


H 





CD 














M-I 


H 














O 


















cd 


H CM 


CO 


-st 


m 








*rl 














d 
















S3 


Va 















DISTRIBUTION LIST 



NAVY 

4 Dr. Marshall J- Farr, Director 
Personnel & Training Research 

Programs 
Office of Naval Research 
Arlington, VA 22217 

1 Director 

ONR Branch Office 
495 Summer Street 
Boston, MA 02210 
ATTN: C. M. Harsh 

1 Director 

ONR Branch Office 
1030 East Green Street 
Pasadena, CA 91101 
ATTN: Dr. E. E". Gloye 

1 Director • 

ONR Branch Office 
536 South Clark Street 
Chicago, IL 60605 
ATTN: M. A. Bertin 

6 Director 

Naval Research Laboratory 
Code 2627 

Washington, DC 20390 

12 Defense Documentation Center 
Cameron Station, Building 5 
5010 Duke Street 
Alexandria, VA 22314 



1 Dr. Harold Booher 
NAVAIR 415C 

Naval Mr Systems Command 
5600 Columbia Pike 
F ; alls Church, VA 22042 

1 Special Assistant for Manpower 
0ASN (M&RA) 

The Pentagon, Room 4E794 
Washington, DC 2035^ 

1 Dr. Richard J. Niehaus 

Office of Civilian Manpower 

Management 
Code 06A 

Department of the Navy 
Washington, DC 20390 

1. CDR Richard L, Martin, USN 
COMFAIRMIRAMAR F-14 
NAS Miramar, CA 92145 

1 Research Director, Code 06 
Research and Evaluation 

Department 
U.S. Naval Examining Center 
Great Lakes, IL 60088 
ATTN: C, Winiewicz 

1 Commanding Off icer 

Naval Medical Neuropsychiatric 

Research Unit 
San Diego, CA 92152 



1 Chief of Naval Technical draining 
Naval Air Station Memphis (75) 
Millington, TN 38054 
ATTN: Dr. G. D. Mayo 



1 Technical Reference Library 

Naval Medical Research Institute 
National Naval Medical Center 
Bethesda, MD 20014 



Chief of Naval Training 
Naval Air Station 
Pensacola, FL 32508 
ATTN: Mr. Ray Williams 



Mr. Lee Miller (AIR 413E) 
Naval Air Systems Command 
5600 Columbia Pike 
Fails Church, VA 22042 




1 Chief 

Bureau of Medicine and Surgery 
Research Division (Code 713) 
Department of the Navy 
Washington, DC 20372 



260 



1 Dr. John J. Collins 

Chief of Naval Operations (OP-987F) 
Department of the Navy 
Washington, DC 20350 

1 Technical Library 

Bureau of Naval Personnel 
Department of the Navy 
Washington, DC 20360 

1 Head, Personnel Measurement Staff 
Capital Area Personnel Office 
Ballston Tower #2, Room 1204 
801 N. Randolph Street 
Arlington, VA 22203 

1 Technical Director 

Naval Personnel Research and 

Development Center 
San Diego, CA 92152 

1 Dr. Norman Abrahams 

Naval Persor-'^l Research and 

Development Center 
San Diego, CA 92152 

1 Dr. Bernard Rimland 

Naval Perronnel Research and 

Development Center 
San Diego, CA 92152 

1 Superintendent 

Naval Postgraduate School 
Monterey, CA 92940 . 
ATTN: Library (Code 2124) 

1 Technical Library 

Naval Ship Systems Command 
National Center, Building 3 
Room 3S08 

Washington, DC 20360 

1 Dr, William L. Maloy 

Principal Civilian Advisor 

for Education and Training 
Naval Training Command, Code 01A 
Pensacola, FL 32508 

1 Mr- Arnold Rubinstein 

Naval Material Command (NMAT-03424) 
Room 820, Crystal Plaza #6 
Washington, DC 20360 



ARMY 

1 U.S. Army Research Institute 
Commonwealth Building, Room 239 
1300 Wilson Boulevard 
Arlington, VA 22209 
ATTN: Dr. R. Dusek 

1 Mr. Edmund F. Fuchs 

U.S. Army Research Institute 
1300 Wilson Boulevard 
Arlington, VA 22209 

1 Dr. Stanley L. Cohen 
Work Unit Area Leader 
Organizational Development 

Work Unit 
Army Res earch Institute for 

Behavioral and Social Science 
1300 Wilson Boulevard 
Arlington, VA 22209 

AIR FORCE 

1 Research and Analysis Division 
AF/DPXYR Room 4C200 
Washington, DC 20330 

1 AFHRL/MD 

701 Prince Street 
Room 200 

Alexandria, VA 22314 

1 Personnel Research Division 
AFHRL 

Lackland Air Force Base 
Texas 78236 

1 AFOSR(NL) 

1400 Wilson Boulevard 
Arlington, VA 22209 

MARINE CORPS 

1 Commandant, Marine Corps 
Code A01M-2 . 
Washington, DC 20380 

1 Dr. A. L. SlafV.osky 

Scientific Advisor (Code Ax) 
Commandant c L * the Marine Corps 
Washington, DC 20380 



261 



i COL George Caridakis 

Director, Office of Manpower 

Utilization 
Headquarters, Marine Corps (A01H) 
MCB 

Quantico, VA 22134 
OTHER POD 

1 Lt. Col. Austin W. Kibler, Director 
Human Resources Research Office 
Advanced Research Projects Agency 
1400 Wilson Boulevard 
Arlington, VA 22209 

1 Dr. Ralph R. Canter 

Director for Manpower Research 
Office of Secretary of Defense 
The Pentagon , Room 3C980 
Washington. DC 20301 

OTHER GOVERNMENT 

1 Dr. Lorraine D. Eyde 

Personnel Research and Development 
Center 

U.S* Civil Service Commission, 

Room 3458 
1900 E. Street, N.W. 
Washington, DC 20415 

• 1 Dr. Vern Urry 

Personnel Research and Development 
Center 

U.S. Civil Service Commission 
Washington, DC 20415 

MISCELLANEOUS . 

1 Dr. Richard C. Atkinson 
Stanford University 
Department of Psychology 
Stanford, CA 94305 

1 Century Research Corporation 

r Lee Highway 

Arlington, VA 22207 

1 Dr. Rene 1 V. Dawis 

University of Minnesota 
Department of Psychology 
Minneapolis, MN 55455 



1 Dr. Robert Dubin 

University of California 
Graduate School of Administration 
Irvine, CA 92664 

1 Dr. Marvin D. Dunnette 
University of Minnesota 
Department of Psychology 
N492 Elliott Hall 
Minneapolis, MN 55455 

2 ERIC 

Processing and Reference Facility 
4833 Rugby Avenue 
Bethesda, MD 20014 ' 

1 Dr. Victor Fields 

Department of Psychology 

Montgomery College 

Rockville, MD 20850 

1 Dr. Edwin A. Fleishman 

American Institutes for Research 
8555 Sixteenth Street 
Silver Spring, MD 20910 

1 Dr. Harry H, Harman 

Educational Testing Service 
Division of Anal 1 ; iical Studies 

and Services 
Princeton, NJ 08540 

1 Dr. Richard S, Hatch 

Decision Systems Associates, Inc. 
11428 Rockville Pike 
Rockville, MD 20852 

1 Human Resources Research 
Organization 
Division #3 
P.O. Box 5787 

Presidio of Monterey, CA 93940 

1 Dr. Lawrence B. Johnson 
Lawrence Johnson and 

Associates, Inc. 
200 S Street, N.W. , Suite 502 
Washington, DC 20009 

1 Dr. Frederick M. Lord 

Educational Testing Service 
Princeton, NJ 08 q /'0 



262 



1 Dr. E. J. McCormick 
Purdue University 
Department of Psychological 

Sciences 
Lafayette, IN 47907 

1 Dr. Joseph W. Rigney 

Behavioral Technology Laboratories 
University of Southern California 
3717 South Grand 
Los Angeles, CA 90007 



1 Dr. Leonard L. Rosenbaum, Chairman 
Department of Psychology 
Montgomery College 
Rockville, MD 20850 

1 Dr. Arthur I. Siegel 

Applied Psychological Services 
Science Center \ 
404 East Lancaster Avenue 
Wayne, PA 19087 

1 Dr. David J. Weiss 

University of Minnesota 
Department of Psychology 
Minneapolis, MN 55455 



9 

ERLC 



